diff --git a/docs/adr/0033-declare-a-canonical-plus-copies-cluster-as-one-arrow-line.md b/docs/adr/0033-declare-a-canonical-plus-copies-cluster-as-one-arrow-line.md new file mode 100644 index 0000000000..934c700f87 --- /dev/null +++ b/docs/adr/0033-declare-a-canonical-plus-copies-cluster-as-one-arrow-line.md @@ -0,0 +1,40 @@ +# Declare a canonical-plus-copies cluster as one arrow line + +- Status: accepted +- Date: 2026-09-11 + +## Context + +`scripts/cross-plugin-source-registry.txt` declares the replication this +repository does on purpose, one path-within-plugin per line, and three readers +key on that shape: the drift checker (`scripts/check-cross-plugin-source-drift.sh`), +the code-metrics replica collapser (`replica-collapse.py`), and the +duplication audit's exclusion filter (`registry-filter.py`). A path-within-plugin +cannot name a canonical copy that lives outside every plugin, so the eighteen +byte-identical copies of `hook-utils.sh` (root `lib/` plus seventeen plugins) +survived the duplication audit as one class with eighteen instances: the +seventeen plugin copies matched the line and the root copy did not. + +## Decision + +**A registry line containing ` -> ` is a cluster line:** the text before the +arrow is the root-relative canonical copy, the whitespace-separated tokens +after it are the members, each a literal root-relative path or a gitignore-style +glob (`lib/hook-utils.sh -> plugins/*/hooks/hook-utils.sh`). The duplication +filter excludes a clone class when every instance is the canonical or matches a +member and the instances sit in pairwise distinct directories; the drift +checker and the replica collapser skip the line, because they key clusters by +path-within-plugin and a root path is not one. A plain line keeps its meaning, +taken whole with any spaces. Lines are tried in file order and the first match +wins. + +## Why + +The registry is a contract every reader parses, so its grammar is hard to +change once lines exist. Splitting on whitespace was rejected: the drift +checker deliberately protects a registered path that contains a space, and a +second file or a YAML registry would have doubled the surface every reader +resolves. A marker that cannot occur in a path-within-plugin (` -> `) lets the +readers that do not understand a cluster ignore it with one test and lets the +one reader that does carry the whole class, canonical included, as a single +exclusion the report names by its line. diff --git a/plugins/claude-config/.claude-plugin/plugin.json b/plugins/claude-config/.claude-plugin/plugin.json index 8eea1a1bee..78c0bd3154 100644 --- a/plugins/claude-config/.claude-plugin/plugin.json +++ b/plugins/claude-config/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "claude-config", - "version": "0.42.3", + "version": "0.42.4", "description": "Nine configuration-health skills (plus setup) for a repo's Claude Code configuration: audit (settings.json / .mcp.json / hooks / plugins / permissions drift), audit-automation-gaps (evidence-gated verdicts on automation gaps), audit-permission-grants (allow-rule / allowed-tools grants for auto-mode durability and portability), audit-permission-state (the permission rules actually in effect: every settings scope merged with per-rule provenance, what auto mode drops on entry, config written where nothing reads it, and which managed intents are enforced versus loosenable), draft-auto-mode-rules (interview and draft a paste-ready autoMode classifier block; prints only, never writes), audit-instructions (locally-owned instruction surfaces vs current model capability, proposing removals/rewrites of instructions the model no longer needs, and detecting cross-surface instruction conflicts), audit-prompting-postures (the additive lane: posture guidance the prompting guide says a component's purpose needs but the component does not carry), audit-pass (one coordinated, ordered, resumable pass over a named target: three-scope inventory, run-time-derived exclusion set, stable finding identity, suppression memory, resume, one human gate, delegating every check to the plugin that owns it), and unhobble (the empirical bare-baseline experiment: reversibly strip a repo's standing instructions, log real stumbles against the current model, re-add only what evidence earns).", "author": { "name": "Melodic Software", diff --git a/plugins/claude-config/CHANGELOG.md b/plugins/claude-config/CHANGELOG.md index 4a5d3264c5..8bb5aed5f2 100644 --- a/plugins/claude-config/CHANGELOG.md +++ b/plugins/claude-config/CHANGELOG.md @@ -3,6 +3,17 @@ All notable changes to the `claude-config` plugin are documented here. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this plugin uses semantic versioning. +## [0.42.4] + +### Changed + +- **`audit-pass`: the exclusion-set reference describes both registry line shapes.** The + sanctioned-replication cluster class is derived from the marketplace's + `scripts/cross-plugin-source-registry.txt`, whose plain entries are paths within each plugin and + whose cluster lines, ` -> ...`, name a root-relative canonical copy and the + plugin paths or globs that carry it; the reference now says how to resolve each shape to its + copy set. Wording only; the pass's behavior is unchanged. + ## [0.42.3] ### Changed diff --git a/plugins/claude-config/skills/audit-pass/reference/exclusion-set.md b/plugins/claude-config/skills/audit-pass/reference/exclusion-set.md index fcbca34bc4..28b36e7251 100644 --- a/plugins/claude-config/skills/audit-pass/reference/exclusion-set.md +++ b/plugins/claude-config/skills/audit-pass/reference/exclusion-set.md @@ -15,8 +15,10 @@ dedicated script. Editing one copy breaks the sync path; a fix-capable pass that corrupt the cluster. **Derivation.** Ask the target whether it documents a shared-source registry. In this marketplace -that is `scripts/cross-plugin-source-registry.txt`, whose entries are paths *within* each plugin; -resolve each entry against every plugin root to get the live copy set. When the target documents no +that is `scripts/cross-plugin-source-registry.txt`, whose plain entries are paths *within* each +plugin (resolve each against every plugin root to get the live copy set) and whose cluster lines, +` -> ...`, name a root-relative canonical copy and the plugin paths or globs that +carry it (the canonical and every match are the copy set). When the target documents no such registry, **this class is empty**. Say so in `skipped` rather than inferring one from similarity, which would exclude files nobody registered. diff --git a/plugins/code-metrics/.claude-plugin/plugin.json b/plugins/code-metrics/.claude-plugin/plugin.json index 2e2645f28d..a9ddcd43de 100644 --- a/plugins/code-metrics/.claude-plugin/plugin.json +++ b/plugins/code-metrics/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "code-metrics", - "version": "0.3.0", + "version": "0.3.1", "description": "Read-only code measures for a change, with cited references and no verdict: lines per file (audit-size), cyclomatic, cognitive, and Halstead complexity (audit-complexity), duplication with sanctioned-replication exclusions (audit-duplication), coverage per function with CRAP from existing lcov, Cobertura, coverage.py, or Go artifacts (audit-coverage), type debt for TypeScript and Python (audit-type-debt), the literacy router for what each number can and cannot say (principles), and a setup skill for the consumer's .claude/code-metrics.yaml. Runs external collectors only when they already resolve, never installs, never runs tests, never emits a finding.", "author": { "name": "Melodic Software", diff --git a/plugins/code-metrics/CHANGELOG.md b/plugins/code-metrics/CHANGELOG.md index d13c214f33..94afd31fa5 100644 --- a/plugins/code-metrics/CHANGELOG.md +++ b/plugins/code-metrics/CHANGELOG.md @@ -3,6 +3,56 @@ All notable changes to the `code-metrics` plugin are documented here. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this plugin uses semantic versioning. +## [0.3.1] + +### Added + +- **`audit-duplication` merges detector pairs into clone classes.** `jscpd` and PMD CPD report a + clone as a pair, so N copies of one fragment arrived as N-1 rows and the summary counted the + fragment's lines N-1 times. A post-pass (`cluster-clones.py`) now joins rows that share an + instance with an identical file and line range into one row per class, the instances sorted by + path and the row labelled `clustered`; the lines count once. The merge joins on identity, not + overlap: a copy that shares only part of a fragment stays its own group. +- **Explicit size and line caps, reported instead of hidden.** `duplication.max_size` (default + `1mb`, binary units) and `duplication.max_lines` (default `null`) are applied by the jscpd + adapter before the tool runs, because jscpd 4 and 5 disagree on their own `--max-size` and + `--max-lines` defaults and on what `0` means, and neither names a skipped file. A skipped file + makes the lane's run row `partial` with the count and the largest file, the document `partial`, + and the markdown summary carries a `Partial:` line. `0` or `null` means no cap. +- **Registry cluster lines.** A sanctioned-replication registry line ` -> ...` + names a root-relative canonical copy and the plugin paths or gitignore-style globs that carry + it, so a canonical file outside any plugin (this repository's `lib/hook-utils.sh`) can declare + its copies; instance paths are compared root-relative and the first matching line wins. A plain + line is still one path-within-plugin taken whole. +- **Per-lane and per-directory rollups.** `summary.by_lane` and `summary.by_directory` (every + ancestor of each class's first instance, cumulative) are additive `code-metrics/v1` fields, + computed after registry exclusion; `duplication.rollup_depth` (default 2) decides how deep the + markdown `## Rollup` section lists. A class is attributed by its first instance after a + root-relative sort, so the rollup reads the same from the repository root and from a + subdirectory. The schema reference states that readers ignore unknown keys. +- **Run rows carry the install hint as a field.** `run[].hint` holds the first install hint a + failed probe produced, apart from the prose reason, so a renderer can print it once. + +### Changed + +- **The duplication markdown reads as a duplication report.** Clone rows are listed largest + first; the summary line is `Files with clones: N.` instead of the size-shaped `Files. Functions. + Over reference.`; an empty exclusion list is stated with its reason; and a run in which no clone + detector resolved for any lane opens with one headline carrying the install hint and + `/code-metrics:setup`. The skill offers that install to the user and never performs it + unprompted. Every other skill's document renders as before. +- **`reference/collectors.md` pins jscpd 5.2.0** and records the 4.x maintenance line (4.3.0), + which the adapter also translates, the binary size grammar, and the token-count difference + between the majors. + +### Fixed + +- **A lane that skipped every file is `partial`, not `empty`**, and the zero floor counts a + `partial` duplication row as measured, so an all-excluded or clone-free lane that skipped a file + still states `duplicated_lines: 0`. +- **A run from a subdirectory matches the same registry lines as a run from the root**, because + instance paths are normalized against the repository root before matching. + ## [0.3.0] ### Added diff --git a/plugins/code-metrics/README.md b/plugins/code-metrics/README.md index 3648139785..55df505186 100644 --- a/plugins/code-metrics/README.md +++ b/plugins/code-metrics/README.md @@ -14,7 +14,7 @@ value to count against, not a bar. |---|---| | `/code-metrics:audit-complexity` | Per-function cyclomatic and cognitive complexity and Halstead difficulty from whichever collector resolves (`lizard`, `radon`, ESLint rules, `gocyclo`, `gocognit`, `shellmetrics`, `multimetric`), beside the ISO/IEC 5055 §8.2.117 reference of 20 with 10 and 15 selectable; cognitive and Halstead carry no standard threshold. | | `/code-metrics:audit-size` | Lines per file (total, blank, comment, code through `scc`; total and non-blank from a bundled counter otherwise) beside a cited reference; `size.mode: iso-8.2.115` adds the ISO function-percentage form. | -| `/code-metrics:audit-duplication` | Clone groups (duplicated lines and tokens, every instance's range) from `jscpd`, `dupl`, or PMD CPD, minus the replication the repository declares in a sanctioned-replication registry, which is an exclusion, not a suppression. | +| `/code-metrics:audit-duplication` | Clone classes (the detector's pairs merged; duplicated lines and tokens, every instance's range) from `jscpd`, `dupl`, or PMD CPD, rolled up per lane and per directory, minus the replication the repository declares in a sanctioned-replication registry (a path-within-plugin, or a `canonical -> copies` cluster line), which is an exclusion, not a suppression. A file over the size cap is reported as skipped, never silently dropped. | | `/code-metrics:audit-coverage` | Line coverage per file and per function read from the artifacts a build already produced (lcov 1.x and 2.2, Cobertura, coverage.py JSON, Go cover profile), plus CRAP per function from the complexity rows; it never runs a test, a missing artifact is a visible warning, and a function with no executable lines reports `null`, never zero. | | `/code-metrics:audit-type-debt` | The typed-code percentage per file and per lane: `type-coverage` for TypeScript, mypy's `--any-exprs-report` for Python; no standard or CWE anchors the measure, so the reference is `null` by design. C# is reported as not applicable. | | `/code-metrics:principles` | Metric literacy: what each measure can and cannot tell you, where every reference value came from, CRAP's corrected provenance, the cross-metric caveats (carried once, here), and gated pointers to the plugins that own mutation score, tautological tests, dead code, coupling, and lint. | @@ -129,7 +129,8 @@ figure for a live session. `scripts/config-defaults.json`. The setup template and the `reference/config.md` key table both are, by a test and by `scripts/check-code-metrics-config-reference.py`; what remains unbound is the number written into a sentence or a small illustrative table, currently `coverage.reference` - in `audit-coverage`, `duplication.min_tokens` and `duplication.min_lines` in `audit-duplication`, + in `audit-coverage`, `duplication.min_tokens`, `duplication.min_lines`, `duplication.max_size`, + `duplication.max_lines`, and `duplication.rollup_depth` in `audit-duplication`, `type_debt.reference` in `audit-type-debt`, and the cyclomatic reference in `setup`. Those drift silently until someone reads them. diff --git a/plugins/code-metrics/reference/collectors.md b/plugins/code-metrics/reference/collectors.md index 79174a53ef..4b24d9fd0c 100644 --- a/plugins/code-metrics/reference/collectors.md +++ b/plugins/code-metrics/reference/collectors.md @@ -33,7 +33,9 @@ built them could not run the tool; the first live run is that row's recheck trig | Tool or format | Lane(s) | Measure | Claim the adapter relies on | Basis | Verified | Recheck trigger | |---|---|---|---|---|---|---| -| `jscpd` 5.1.2 | every lane | `duplication` | v5 is a Rust binary that only writes `/jscpd-report.json`, never stdout, so the adapter runs `--reporters json --output ` and prints the file itself; `duplicates[]` carries `firstFile`/`secondFile` with `start`, `end`, plus `lines` and `tokens`; `--absolute` is required because names are otherwise relative to the common ancestor of the inputs, which collapses two vendored copies sharing a basename; jscpd 4 wrote a different document under the same name | github.com/kucherenko/jscpd, run in this repository (the capture is `scripts/fixtures/tool-output/jscpd.json`) | 2026-09-05 | a jscpd major release, or a change to the report filename or the `duplicates[]` shape | +| `jscpd` 5.2.0 | every lane | `duplication` | v5 is a Rust binary that only writes `/jscpd-report.json`, never stdout, so the adapter runs `--reporters json --output ` and prints the file itself; `duplicates[]` carries `firstFile`/`secondFile` with `start`, `end`, plus `lines` and `tokens`, and 5.2.0 adds a per-duplicate `kind` (`exact`) the adapter does not read; `--absolute` is required because names are otherwise relative to the common ancestor of the inputs, which collapses two vendored copies sharing a basename; `statistics.total.sources` counts token sources, not files, so it is not a skipped-file signal | github.com/kucherenko/jscpd, run in this repository (the capture is `scripts/fixtures/tool-output/jscpd.json`, from 5.2.0) | 2026-09-11 | a jscpd major release, or a change to the report filename or the `duplicates[]` shape | +| `jscpd` 4.3.0 (the 4.x maintenance line) | every lane | `duplication` | v4 is a Node program that writes the same `/jscpd-report.json` with the same `firstFile`/`secondFile`, `start`, `end`, `lines`, and `tokens` keys, so one adapter translates both majors; it tokenizes differently from v5, so a clone count can differ between the majors on the same input, and the plugin never compares counts across a major boundary | github.com/kucherenko/jscpd (`npm install jscpd@4`), run in this repository over the same cluster fixture | 2026-09-11 | a 4.x release that changes the report shape, or the 4.x line being retired upstream | +| `jscpd` size and line caps (both majors) | every lane | `duplication` | the adapter applies `duplication.max_size` and `duplication.max_lines` itself, before jscpd runs, and passes jscpd one above its own bound (or `2147483647` lines and `1099511627776` bytes when there is no cap, bounds no real file reaches), because the majors disagree on the flags: 4.x defaults to 100kb and 1000 lines and reads `--max-lines 0` as that default, 5.x defaults to 1mb with no line cap, both read `--max-size 0` as skip every file, and neither names a skipped file in the report; jscpd's size grammar is binary (`1kb` is 1,024 bytes, `1mb` is 1,048,576) and the adapter uses the same multipliers | github.com/kucherenko/jscpd, both majors run in this repository with `--max-size` and `--max-lines` set to `0`, the default, and one byte or line below a fixture file's size | 2026-09-11 | either major changes a `--max-size`/`--max-lines` default or the meaning of `0`, or a report gains a skipped-file list | | PMD CPD 7.27.0 | typescript, python, go, dotnet | `duplication` | `pmd cpd --minimum-tokens N --format xml --language --file-list ` (one path per line) prints a namespaced `pmd-cpd` document whose `duplication` elements carry `lines` and `tokens` with one `file` child per instance (`path`, `line`, `endline`); CPD has no JSON reporter, no minimum-lines option, no ignore-glob option, and no Bash or shell language; exit 4 means duplications were found, not that the run failed | docs.pmd-code.org CPD user documentation, CLI reference, and report formats; the adapter and its fixture are unverified against a live run | 2026-09-05 | a PMD 8 release, a JSON reporter, a shell CPD language, or the first live run of this adapter | | `dupl` v1.1.0 | go | `duplication` | the default text printer emits `found clones:` per group, then an indented `:,` line per instance, then a total footer; `-plumbing` is pairwise and loses groups of three or more, so the text printer is parsed; `-t` is a token threshold with no line equivalent; dupl reports no token count and ships no version flag | github.com/mibk/dupl `printer/text.go` and `main.go`; the adapter and its fixture are unverified against a live run | 2026-09-05 | a dupl release that changes the printer, adds a version flag, or adds a token count | diff --git a/plugins/code-metrics/reference/config.md b/plugins/code-metrics/reference/config.md index 8751fd8654..5f71d90321 100644 --- a/plugins/code-metrics/reference/config.md +++ b/plugins/code-metrics/reference/config.md @@ -59,7 +59,7 @@ The third column is written by hand and is not derived from anything. A row whos | `scope.default` | `change` | `change` (the merge-base diff plus uncommitted and untracked files) or `all` | | `scope.base` | `auto` | The merge-base is taken against the default branch, or against this ref | | `scope.exclude` | `["**/node_modules/**", "**/vendor/**", "**/dist/**", "**/build/**"]` | Gitignore-style globs dropped from every measure. The default names dependency and build-output directories, because measuring a compiled bundle or a vendored tree counts functions nobody in the repository wrote; fixtures and evals stay in scope. A closed list, so a team file that sets it replaces the default whole (`[]` measures everything). The count is reported as `scope.excluded` and each pattern's count as `scope.exclusions[]` | -| `scope.registries` | `[]` | Sanctioned-replication registries (one path-within-plugin per line, relative to the repository root), applied by every audit: rows for a file the registry names collapse to one row carrying a replica count, and a clone whose every instance sits at a listed path is excluded, not suppressed | +| `scope.registries` | `[]` | Sanctioned-replication registries, each relative to the repository root, applied by every audit. A plain line is one path-within-plugin, taken whole, spaces included: rows for a file it names collapse to one row carrying a replica count, and a clone is excluded, not suppressed, when every instance ends with that path and the copies sit in distinct carrying directories. A line ` -> ...` is a cluster line read by the duplication audit only: the root-relative canonical copy, then the paths or gitignore-style globs that carry it; a clone is excluded when every instance is the canonical or matches a member and the instances' directories are pairwise distinct. Instance paths are compared root-relative, and the first matching line in file order wins | | `complexity.cyclomatic.reference` | `20` | ISO/IEC 5055:2021 §8.2.117 (normative). Cited alternatives: 10 (McCabe 1976, "reasonable, but not magical") and 15 (NIST SP 500-235, with its six practices) | | `complexity.cognitive.reference` | `null` | Campbell, SonarSource; no standard sets a threshold | | `complexity.halstead.difficulty` | `null` | Halstead 1977; no standard sets a threshold | @@ -69,6 +69,9 @@ The third column is written by hand and is not derived from anything. A row whos | `duplication.min_tokens` | `50` | Passed to the clone collector | | `duplication.min_lines` | `5` | Passed to the clone collector | | `duplication.ignore` | `[]` | Collector ignore globs | +| `duplication.max_lines` | `null` | A file with more lines is left out of the clone scan and named in the lane's `partial` run row; `null` or `0` means no line cap, which is what jscpd 5, PMD CPD, and SonarQube ship. A number is a plugin-local guard, not an upstream convention | +| `duplication.max_size` | `1mb` | A file larger than this is left out and named the same way; `0` means no cap. jscpd 5.0.7 sets 1mb as its parser guard and SonarJS 1000kb for generated code. `kb` and `mb` are binary (1mb is 1,048,576 bytes); a CRLF checkout counts one more byte per line | +| `duplication.rollup_depth` | `2` | Directory depth to which the markdown report lists per-directory rollup rows; the JSON carries every directory | | `duplication.registries` | `[]` | The older name for `scope.registries`, read only when the scope-level list is empty; a team file written against it keeps working unchanged | | `coverage.artifacts` | `[]` | Explicit coverage artifact paths; empty means auto-discover. An explicitly named path that does not exist is a usage error | | `coverage.path_prefix_strip` | `[]` | Prefixes removed from artifact paths before the join with source paths (compiled-output layouts) | diff --git a/plugins/code-metrics/reference/report-schema.md b/plugins/code-metrics/reference/report-schema.md index 19e9186bda..5eba201ad0 100644 --- a/plugins/code-metrics/reference/report-schema.md +++ b/plugins/code-metrics/reference/report-schema.md @@ -11,15 +11,31 @@ read, so its shape is stable within the `v1` schema string. | `schema` | string | `code-metrics/v1` | | `skill` | string | The producing skill, for example `audit-size` | | `generated_at` | string | UTC timestamp, `YYYY-MM-DDTHH:MM:SSZ` | -| `status` | string | `complete` (every implied lane and measure ran; a `not-applicable` row implies nothing and never withholds it), `partial` (at least one `unavailable`, `deferred`, or `partial` row), `empty` (nothing was measured; the markdown headline reads "Measured nothing") | +| `status` | string | `complete` (every implied lane and measure ran; a `not-applicable` row implies nothing and never withholds it), `partial` (at least one `unavailable`, `deferred`, or `partial` row; a lane that skipped every file is `partial` even with no measure row, because its run row states the skip), `empty` (nothing was measured; the markdown headline reads "Measured nothing") | | `scope` | object | `mode` (`change`, `paths`, `all`), `base` (the merge-base's short SHA under `change`, else `null`), `files` (count in scope), `unclassified` (how many of those belong to no lane, so `files` minus `unclassified` is the measured count; with the catch-all `other` lane enabled this is a disabled lane's files, and otherwise 0), `excluded` (count dropped by scope exclusions), `exclusions` (one `{pattern, files}` per `scope.exclude` glob that matched at least one file; a file two globs match counts under both) | | `run` | array | The "Coverage of this run" table, one row per lane and measure the scope implied | | `thresholds` | array | The references in force: `measure`, `value_key` (the `values` key the reference is applied to), `direction` (`at_or_above`, or `below` for coverage and type coverage), `reference` (number or `null`), `provenance`, `layer` (which config layer supplied it, or `bundled default`). The markdown Measures table lists rows over a reference first, furthest past it at the top, then the rest by the first entry whose `value_key` the rows carry, largest first (smallest first under `below`), and its 200-row cap names that key | | `measures` | array | The rows, see below | -| `summary` | object | `files`, `functions`, `over_reference` (measure name to count); when clone-group rows are present, `duplicated_lines` (sum of each group's `values.lines`, one group counted once, after registry exclusions) and `clone_groups` | -| `excluded` | array | Duplication only: clone groups dropped by a sanctioned-replication registry, each naming the registry path and line | +| `summary` | object | `files`, `functions`, `over_reference` (measure name to count); when clone-group rows are present, `duplicated_lines` (sum of each group's `values.lines`, one group counted once, after registry exclusions), `clone_groups`, `by_lane`, and `by_directory` (see below) | +| `excluded` | array | Duplication only: clone groups dropped by a sanctioned-replication registry (intentional clones the repository declares about itself), each naming the registry path and line | | `unavailable` | array | `lane/measure` strings for every `run` row whose status is `unavailable` | +A reader ignores keys it does not know: fields are added within `v1` (the rollups and the run +row's `hint` were), never renamed or removed. + +## Duplication rollups + +`summary.by_lane` maps each lane to `{"groups", "duplicated_lines"}` over the surviving clone +groups, and `summary.by_directory` maps `.` and every ancestor directory of each group's first +instance (instances are sorted by path, so that is the lowest path; paths are made relative to the +repository root) to the same shape. A group counts once under every ancestor, so a parent includes +its children and directory rows cannot be summed; two identities hold instead: +`by_directory["."].duplicated_lines == summary.duplicated_lines` and the `by_lane` values sum to +it. A duplication run that found or kept no group carries both as empty maps, beside its +`duplicated_lines: 0`; a document without clone-group rows and without a duplication collector run +carries neither. The markdown `## Rollup` section lists directories to `duplication.rollup_depth` +(default 2); the JSON carries every directory. + `summary.functions` counts functions, not rows: one function measured by two collectors produces two rows and counts once. Rows are grouped by file and name, and a group counts as many functions as it has distinct `start_line` values, or as one when no row in it reports a start line. So two `render` @@ -47,17 +63,21 @@ different files whatever the registry says, and all of them stay. Clone-group ro `lane`, `measure`, `collector` (the tool and version that produced the rows, or `null`), `status` (`ok`, `partial`, `unavailable`, `not-applicable`, `deferred`), `reason` (`null` only when `ok` and the collector said nothing; an `ok` row whose collector wrote to stderr while succeeding carries -that text, such as mypy's `mypy reported 386 errors (349 missing stubs)`). A -run whose scope holds no measurable file carries one row `*/*` with status `not-applicable` and a -reason that opens with `no measurable files in scope` and, under `change`, says why: the branch is -at its merge-base with a clean working tree (naming the ref and the `--all` alternative), or the +that text, such as mypy's `mypy reported 386 errors (349 missing stubs)`), `hint` (the first +install hint a failed probe produced for the row, or `null`; it is kept apart from the prose +reason so a renderer can print it once without parsing it back out). A run whose scope holds no +measurable file carries one row `*/*` with status `not-applicable` and a reason that opens with +`no measurable files in scope` and, under `change`, says why: the branch is at its merge-base with +a clean working tree (naming the ref and the `--all` alternative), or the changed files belong to no lane. The lane `other` (every text file outside the language lanes) has `file_lines` rows only; each other measure carries a `not-applicable` row for it. `partial` means the row produced measurements for some of what it implied and not the rest, which `audit-coverage` emits when an artifact covers only some of a lane's scope files, and again when it -left a function unjoined, naming those functions in the reason. It counts as having produced rows, -so such a run is `partial` rather than `empty`, and it withholds `complete`, so a document can never +left a function unjoined, naming those functions in the reason, and which `audit-duplication` emits +when a `duplication.max_size` or `duplication.max_lines` cap left files out of the clone scan, +naming the count and the largest skipped file in the reason. It counts as having produced rows, so +such a run is `partial` rather than `empty`, and it withholds `complete`, so a document can never read as complete while one of its own rows says `N of M`. ## `measures[]` rows diff --git a/plugins/code-metrics/scripts/collectors/jscpd.py b/plugins/code-metrics/scripts/collectors/jscpd.py index be6f955b68..2357219b44 100755 --- a/plugins/code-metrics/scripts/collectors/jscpd.py +++ b/plugins/code-metrics/scripts/collectors/jscpd.py @@ -4,15 +4,20 @@ Adapter contract (design/contracts.md section 3): `probe`, `measures`, `collect ...`, `install_hint`. -jscpd 5 is a Rust binary that only writes its report to a file, so `collect` -runs it with `--reporters json --output `, reads -`/jscpd-report.json`, prints the translated rows, and deletes the -temporary directory (probed 2026-09-05 against jscpd 5.1.2). `--absolute` is -passed because jscpd otherwise names files relative to the common ancestor of -its inputs, which collapses two vendored copies that share a basename into one -indistinguishable name; the absolute paths are made relative to the working -directory here. jscpd 4 wrote a different document under the same name and is -not translated by this file. +jscpd only writes its report to a file, so `collect` runs it with +`--reporters json --output `, reads `/jscpd-report.json`, +prints the translated rows, and deletes the temporary directory. Both +maintained majors are translated: the 4.x line (Node) and the 5.x line (Rust) +write `duplicates[]` entries with `firstFile`/`secondFile` under the same +report name, and this adapter reads only those keys plus `lines` and `tokens` +(verified 2026-09-11 against jscpd 4.3.0 and 5.2.0; recheck when a major above +5 ships). 5.2.0 adds a per-duplicate `kind` (`exact`) that is not read, and +the two majors tokenize differently, so a clone count can differ between them +on the same input. +`--absolute` is passed because jscpd otherwise names files relative to the +common ancestor of its inputs, which collapses two vendored copies that share +a basename into one indistinguishable name; the absolute paths are made +relative to the working directory here. Each duplicate becomes one clone-group row: `file` and `function` are null, `instances[]` carries every copy with its line range, and `values` carries @@ -22,6 +27,23 @@ CODE_METRICS_DUP_MIN_TOKENS jscpd --min-tokens (default 50) CODE_METRICS_DUP_MIN_LINES jscpd --min-lines (default 5) CODE_METRICS_DUP_IGNORE jscpd --ignore, comma-separated globs (default none) + CODE_METRICS_DUP_MAX_SIZE files larger than this are skipped (default 1mb; + empty or 0 means no cap; kb/mb/gb are binary) + CODE_METRICS_DUP_MAX_LINES files with more lines are skipped (default none; + empty or 0 means no cap) + +The caps are applied HERE, before jscpd runs, and not delegated to jscpd's +own `--max-size`/`--max-lines`: the two majors disagree on what those flags +default to (4.x caps at 1000 lines and 100kb, 5.x at 1mb and no line cap), +on what `0` means (4.x reads `--max-lines 0` as the default and `--max-size 0` +as "skip everything"), and neither names a skipped file in the report, so a +file left out of the scan would be invisible. jscpd is passed one more than +the adapter's own bound (or a bound no real file reaches when there is no +cap) so the pre-filter is the only gate on either major. Every skip is +reported as one line to the file named by CODE_METRICS_PARTIAL_REASON_FILE +(the dispatcher's channel for a `partial` run row), or to stderr when that +variable is unset. `statistics.total.sources` in the jscpd report counts token +sources, not files, and is not read for this purpose. jscpd's own exit code is not read: it exits non-zero when a `--threshold` or `--exit-code` run finds clones, and this adapter passes neither, so the report @@ -46,6 +68,15 @@ REPORT_BASENAME = "jscpd-report.json" DEFAULT_MIN_TOKENS = "50" DEFAULT_MIN_LINES = "5" +DEFAULT_MAX_SIZE = "1mb" +# Passed to jscpd when the adapter applies no cap of its own: bounds no real +# file reaches, so jscpd's own gate can never skip a file this adapter did not +# name; both are accepted by both majors (`0` is not), verified 2026-09-11 +# against 4.3.0 and 5.2.0 with the line bound at the signed 32-bit maximum. +NO_LINE_CAP = 2_147_483_647 +NO_SIZE_CAP = 1 << 40 +_SIZE_UNITS = {"": 1, "b": 1, "kb": 1024, "mb": 1024**2, "gb": 1024**3} +_SIZE_RE = re.compile(r"^(\d+(?:\.\d+)?)\s*([kmg]?b)?$") def _normalize(path: str) -> str: @@ -87,6 +118,114 @@ def probe() -> int: return 0 +def parse_size(text: str) -> int | None: + """Bytes for a size such as `1mb`, `100kb`, `2048`; None for no cap. + + Units are binary (1kb = 1024 bytes), the multiplier jscpd's own grammar + uses. Empty and `0` mean no cap. Anything else raises ValueError. + """ + text = (text or "").strip().lower() + if not text: + return None + match = _SIZE_RE.match(text) + if not match: + raise ValueError(f"not a size: {text!r} (expected e.g. 1mb, 100kb, 2048)") + value = int(float(match.group(1)) * _SIZE_UNITS[match.group(2) or ""]) + return value if value > 0 else None + + +def parse_lines(text: str) -> int | None: + """A positive line cap, or None for no cap (empty, 0, or negative).""" + text = (text or "").strip() + if not text: + return None + try: + value = int(text) + except ValueError as exc: + raise ValueError(f"not a line count: {text!r}") from exc + return value if value > 0 else None + + +def count_lines(path: str) -> int: + """Newline count, plus one for a final line without a newline.""" + lines = 0 + last = b"\n" + with open(path, "rb") as handle: + for chunk in iter(lambda: handle.read(1 << 16), b""): + lines += chunk.count(b"\n") + last = chunk[-1:] + if last != b"\n": + lines += 1 + return lines + + +def prefilter( + files: list[str], size_cap: int | None, line_cap: int | None +) -> tuple[list[str], list[tuple[str, int, int | None]]]: + """Split `files` into the ones jscpd scans and the ones a cap skips. + + A skipped entry is `(path, bytes, lines)`; `lines` is None when no line cap + is set, because counting lines reads the whole file and the size cap + needs only a stat. A file that cannot be stat'ed is kept, so jscpd (and its + own error message) decides what to do with it. + """ + kept: list[str] = [] + skipped: list[tuple[str, int, int | None]] = [] + for path in files: + try: + size = os.stat(path).st_size + except OSError: + kept.append(path) + continue + lines: int | None = None + if size_cap is not None and size > size_cap: + skipped.append((path, size, None)) + continue + if line_cap is not None: + try: + lines = count_lines(path) + except OSError: + kept.append(path) + continue + if lines > line_cap: + skipped.append((path, size, lines)) + continue + kept.append(path) + return kept, skipped + + +def report_skips( + skipped: list[tuple[str, int, int | None]], + total: int, + size_text: str, + lines_text: str, +) -> None: + """One line naming how many files a cap left out, and the largest one.""" + if not skipped: + return + largest = max(skipped, key=lambda entry: entry[1]) + path, size, lines = largest + if lines is None: + try: + lines = count_lines(path) + except OSError: + lines = 0 + reason = ( + f"{len(skipped)} of {total} files skipped by duplication.max_size " + f"{size_text or 'none'} / max_lines {lines_text or 'none'}; " + f"largest: {_normalize(path)} ({size} bytes, {lines} lines)" + ) + target = os.environ.get("CODE_METRICS_PARTIAL_REASON_FILE") or "" + if target: + try: + with open(target, "w", encoding="utf-8") as handle: + handle.write(reason + "\n") + return + except OSError as exc: + print(f"jscpd.py: cannot write {target}: {exc}", file=sys.stderr) + print(f"jscpd.py: {reason}", file=sys.stderr) + + def _instance(entry: dict) -> dict: return { "file": _normalize(str(entry.get("name", ""))), @@ -123,7 +262,13 @@ def translate(raw: str, lane: str) -> list[dict]: return rows -def _command(exe: str, output: str, files: list[str]) -> list[str]: +def _command( + exe: str, + output: str, + files: list[str], + size_cap: int | None = None, + line_cap: int | None = None, +) -> list[str]: command = [ exe, "--reporters", @@ -134,6 +279,12 @@ def _command(exe: str, output: str, files: list[str]) -> list[str]: os.environ.get("CODE_METRICS_DUP_MIN_TOKENS") or DEFAULT_MIN_TOKENS, "--min-lines", os.environ.get("CODE_METRICS_DUP_MIN_LINES") or DEFAULT_MIN_LINES, + # One above the adapter's own bound: the pre-filter already removed + # every file over it, so jscpd's gate never fires on either major. + "--max-size", + str(size_cap + 1 if size_cap is not None else NO_SIZE_CAP), + "--max-lines", + str(line_cap + 1 if line_cap is not None else NO_LINE_CAP), "--absolute", "--silent", ] @@ -151,10 +302,24 @@ def collect(lane: str, measure: str, files: list[str]) -> int: if not exe: print("jscpd not on PATH or in ./node_modules/.bin", file=sys.stderr) return 3 + size_text = os.environ.get("CODE_METRICS_DUP_MAX_SIZE", DEFAULT_MAX_SIZE) + lines_text = os.environ.get("CODE_METRICS_DUP_MAX_LINES", "") + try: + size_cap = parse_size(size_text) + line_cap = parse_lines(lines_text) + except ValueError as exc: + print(f"jscpd.py: {exc}", file=sys.stderr) + return 2 + kept, skipped = prefilter(files, size_cap, line_cap) + report_skips(skipped, len(files), size_text.strip(), lines_text.strip()) + if not kept: + # Nothing left to scan is a measurement of zero, not a failure; the + # skip line above says what was left out. + return 0 output = tempfile.mkdtemp(prefix="code-metrics-jscpd-") try: result = subprocess.run( - _command(exe, output, files), + _command(exe, output, kept, size_cap, line_cap), capture_output=True, text=True, check=False, diff --git a/plugins/code-metrics/scripts/collectors/test_jscpd.py b/plugins/code-metrics/scripts/collectors/test_jscpd.py index e057a2b8ae..238f53d1cb 100755 --- a/plugins/code-metrics/scripts/collectors/test_jscpd.py +++ b/plugins/code-metrics/scripts/collectors/test_jscpd.py @@ -6,8 +6,10 @@ copies the committed capture fixtures/tool-output/jscpd.json into the `--output` directory the adapter passes, the way jscpd 5 writes its own report (design T13; no executable is committed). The capture came from a live -jscpd 5.1.2 run over the two-copy cluster under -fixtures/sources/cluster/{alpha,beta}/shared/shared-utils.sh. +jscpd 5.2.0 run over the two-copy cluster under +fixtures/sources/cluster/{alpha,beta}/shared/shared-utils.sh, rewritten to +repo-relative names; 4.3.0 writes the same keys the adapter reads, so one +capture stands in for both majors and the stub only varies the version line. """ from __future__ import annotations @@ -33,7 +35,7 @@ def make_stub( directory: Path, - version_line: str = "jscpd 5.1.2", + version_line: str = "jscpd 5.2.0", capture: Path = CAPTURE, exit_code: int = 0, argv_log: Path | None = None, @@ -90,7 +92,7 @@ def test_probe_prints_the_version_when_the_stub_resolves(self) -> None: with tempfile.TemporaryDirectory() as tmp: make_stub(Path(tmp)) result = run("probe", path_prefix=Path(tmp)) - self.assertEqual((result.returncode, result.stdout.strip()), (0, "5.1.2")) + self.assertEqual((result.returncode, result.stdout.strip()), (0, "5.2.0")) def test_collect_translates_the_capture_into_one_clone_group(self) -> None: with tempfile.TemporaryDirectory() as tmp: @@ -185,7 +187,7 @@ def test_no_report_file_is_exit_3_with_the_tool_stderr(self) -> None: stub = Path(tmp) / "jscpd" stub.write_text( "#!/usr/bin/env bash\n" - 'if [[ "${1:-}" == "--version" ]]; then printf \'jscpd 5.1.2\\n\'; exit 0; fi\n' + 'if [[ "${1:-}" == "--version" ]]; then printf \'jscpd 5.2.0\\n\'; exit 0; fi\n' "printf 'jscpd: unsupported format\\n' >&2\n" "exit 1\n", encoding="utf-8", @@ -204,6 +206,142 @@ def test_the_temporary_output_directory_is_removed(self) -> None: run("collect", "bash", "duplication", ALPHA, BETA, path_prefix=Path(tmp)) self.assertEqual(set(glob.glob(pattern)) - before, set()) + def test_explicit_caps_reach_the_command_line_on_both_majors(self) -> None: + for version in ("jscpd 4.3.0", "jscpd 5.2.0"): + with tempfile.TemporaryDirectory() as tmp: + log = Path(tmp) / "argv.log" + make_stub(Path(tmp), version_line=version, argv_log=log) + result = run( + "collect", + "bash", + "duplication", + ALPHA, + BETA, + path_prefix=Path(tmp), + env_extra={ + "CODE_METRICS_DUP_MAX_SIZE": "1mb", + "CODE_METRICS_DUP_MAX_LINES": "", + }, + ) + self.assertEqual(result.returncode, 0, result.stderr) + argv = log.read_text(encoding="utf-8") + # One byte above the adapter's own bound, so the pre-filter is + # the only gate on either major. + self.assertIn("--max-size 1048577", argv, version) + self.assertIn("--max-lines 2147483647", argv, version) + + def test_a_zero_cap_means_no_cap_and_is_never_passed(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + log = Path(tmp) / "argv.log" + make_stub(Path(tmp), version_line="jscpd 4.3.0", argv_log=log) + result = run( + "collect", + "bash", + "duplication", + ALPHA, + path_prefix=Path(tmp), + env_extra={ + "CODE_METRICS_DUP_MAX_SIZE": "0", + "CODE_METRICS_DUP_MAX_LINES": "0", + }, + ) + self.assertEqual(result.returncode, 0, result.stderr) + argv = log.read_text(encoding="utf-8") + self.assertNotIn("--max-size 0 ", argv + " ") + self.assertNotIn("--max-lines 0 ", argv + " ") + self.assertIn("--max-lines 2147483647", argv) + self.assertIn("--max-size 1099511627776", argv) + + def test_the_size_grammar_uses_binary_multipliers(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + log = Path(tmp) / "argv.log" + make_stub(Path(tmp), argv_log=log) + result = run( + "collect", + "bash", + "duplication", + ALPHA, + path_prefix=Path(tmp), + env_extra={ + "CODE_METRICS_DUP_MAX_SIZE": "2kb", + "CODE_METRICS_DUP_MAX_LINES": "44", + }, + ) + self.assertEqual(result.returncode, 0, result.stderr) + argv = log.read_text(encoding="utf-8") + self.assertIn("--max-size 2049", argv) + self.assertIn("--max-lines 45", argv) + + def test_files_over_the_cap_are_skipped_and_the_reason_is_recorded(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + log = Path(tmp) / "argv.log" + note = Path(tmp) / "partial" + big = Path(tmp) / "big.sh" + big.write_text("echo line\n" * 100, encoding="utf-8") + make_stub(Path(tmp), argv_log=log) + result = run( + "collect", + "bash", + "duplication", + ALPHA, + str(big), + path_prefix=Path(tmp), + env_extra={ + "CODE_METRICS_DUP_MAX_LINES": "50", + "CODE_METRICS_PARTIAL_REASON_FILE": str(note), + }, + ) + self.assertEqual(result.returncode, 0, result.stderr) + argv = log.read_text(encoding="utf-8") + self.assertIn(ALPHA, argv) + self.assertNotIn("big.sh", argv) + self.assertRegex( + note.read_text(encoding="utf-8").strip(), + r"^1 of 2 files skipped by duplication\.max_size 1mb / max_lines 50; " + r"largest: .*big\.sh \(\d+ bytes, 100 lines\)$", + ) + + def test_all_files_skipped_returns_zero_without_invoking_jscpd(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + log = Path(tmp) / "argv.log" + note = Path(tmp) / "partial" + make_stub(Path(tmp), argv_log=log) + result = run( + "collect", + "bash", + "duplication", + ALPHA, + BETA, + path_prefix=Path(tmp), + env_extra={ + "CODE_METRICS_DUP_MAX_SIZE": "10", + "CODE_METRICS_PARTIAL_REASON_FILE": str(note), + }, + ) + self.assertEqual((result.returncode, result.stdout), (0, ""), result.stderr) + self.assertFalse(log.exists(), "jscpd was invoked with no files") + self.assertTrue( + note.read_text(encoding="utf-8").startswith( + "2 of 2 files skipped by duplication.max_size 10 / max_lines none" + ), + note.read_text(encoding="utf-8"), + ) + + def test_the_skip_reason_goes_to_stderr_when_no_reason_file_is_set(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + make_stub(Path(tmp)) + result = run( + "collect", + "bash", + "duplication", + ALPHA, + BETA, + path_prefix=Path(tmp), + env_extra={"CODE_METRICS_DUP_MAX_SIZE": "10"}, + ) + self.assertEqual(result.returncode, 0) + self.assertIn("2 of 2 files skipped", result.stderr) + def test_other_verbs(self) -> None: self.assertEqual(run("measures").stdout.strip(), "*/duplication") self.assertIn("kucherenko/jscpd", run("install_hint").stdout) diff --git a/plugins/code-metrics/scripts/config-defaults.json b/plugins/code-metrics/scripts/config-defaults.json index 61e23fd970..13068e64e6 100644 --- a/plugins/code-metrics/scripts/config-defaults.json +++ b/plugins/code-metrics/scripts/config-defaults.json @@ -19,7 +19,10 @@ "min_tokens": 50, "min_lines": 5, "ignore": [], - "registries": [] + "registries": [], + "max_lines": null, + "max_size": "1mb", + "rollup_depth": 2 }, "coverage": { "artifacts": [], diff --git a/plugins/code-metrics/scripts/dispatch.sh b/plugins/code-metrics/scripts/dispatch.sh index e98846e1b2..d0d556c5ed 100755 --- a/plugins/code-metrics/scripts/dispatch.sh +++ b/plugins/code-metrics/scripts/dispatch.sh @@ -501,20 +501,22 @@ ROWS="$WORK/measures.jsonl" COLLECT_FAILED=0 run_row() { - # run_row + # run_row [] # # One file per slot, concatenated in slot order once every collector has # finished, so the run table reads lane by lane and measure by measure # whatever order the parallel collectors happened to complete in. # # One interpreter call per row rather than one per field; an empty - # collector or reason is null. + # collector, reason, or hint is null. `hint` is the first install hint a + # failed probe produced for the row, kept apart from the prose reason so a + # renderer can print it once without parsing it back out. "${PY[@]}" -c ' import json, sys -lane, measure, collector, status, reason = sys.argv[1:6] +lane, measure, collector, status, reason, hint = sys.argv[1:7] print(json.dumps({"lane": lane, "measure": measure, "collector": collector or None, - "status": status, "reason": reason or None})) -' "$2" "$3" "$4" "$5" "$6" >"$WORK/run.$1" + "status": status, "reason": reason or None, "hint": hint or None})) +' "$2" "$3" "$4" "$5" "$6" "${7:-}" >"$WORK/run.$1" } IFS=',' read -r -a MEASURE_LIST <<<"$MEASURES" @@ -578,7 +580,12 @@ launch_collect() { printf '%s\n' "$@" >"$WORK/files.$slot" ( started="$(date +%s)" - "${PY[@]}" "$adapter" collect "$lane" "$measure" --paths-from "$WORK/files.$slot" >"$WORK/out.$slot" 2>"$WORK/err.$slot" + # A collector that leaves some of its inputs out (a cap it applies + # itself) writes one line to the partial-reason file; a successful + # collect with a non-empty file is a `partial` row carrying that line, + # never a silent `ok`. + CODE_METRICS_PARTIAL_REASON_FILE="$WORK/partial.$slot" \ + "${PY[@]}" "$adapter" collect "$lane" "$measure" --paths-from "$WORK/files.$slot" >"$WORK/out.$slot" 2>"$WORK/err.$slot" rc=$? printf '%s %s\n' "$rc" "$(($(date +%s) - started))" >"$WORK/rc.$slot" ) & @@ -596,6 +603,7 @@ for lane in "${LANES[@]}"; do SLOT=$((SLOT + 1)) resolved=0 reasons="" + first_hint="" while IFS=$'\t' read -r tool note; do [[ -n "$tool" ]] || continue case "$tool" in @@ -629,6 +637,7 @@ for lane in "${LANES[@]}"; do why="$(tr '\n' ' ' <"$probe_err" | cut -c1-200)" why="${why% }" reasons+="${reasons:+; }$tool: ${why:-not found}${hint:+ ($hint)}" + [[ -n "$first_hint" || -z "$hint" ]] || first_hint="$hint" continue fi S_LANE[slot]="$lane" @@ -642,7 +651,7 @@ for lane in "${LANES[@]}"; do break done < <(ladder_tools "$lane" "$measure") if [[ $resolved -eq 0 ]]; then - run_row "$slot" "$lane" "$measure" '' unavailable "${reasons:-no ladder entry for $lane/$measure}" + run_row "$slot" "$lane" "$measure" '' unavailable "${reasons:-no ladder entry for $lane/$measure}" "$first_hint" fi done done @@ -658,11 +667,15 @@ for slot in "${COLLECT_SLOTS[@]}"; do version="${S_VERSION[slot]}" if [[ "${rc:-1}" -eq 0 ]]; then cat "$WORK/out.$slot" >>"$ROWS" - # What an adapter said on stderr while succeeding (mypy's error count, - # a module the scope did not cover) is the ok row's reason; an adapter - # that said nothing leaves it null. - note="$(tr '\n' ' ' <"$WORK/err.$slot" | cut -c1-500)" - run_row "$slot" "$lane" "$measure" "$tool $version" ok "${note% }" + if [[ -s "$WORK/partial.$slot" ]]; then + run_row "$slot" "$lane" "$measure" "$tool $version" partial "$(head -n 1 "$WORK/partial.$slot")" + else + # What an adapter said on stderr while succeeding (mypy's error count, + # a module the scope did not cover) is the ok row's reason; an adapter + # that said nothing leaves it null. + note="$(tr '\n' ' ' <"$WORK/err.$slot" | cut -c1-500)" + run_row "$slot" "$lane" "$measure" "$tool $version" ok "${note% }" + fi progress "$lane/$measure: $tool finished in ${elapsed:-?}s, $(wc -l <"$WORK/out.$slot" | tr -d ' ') row(s)" elif [[ "${rc:-1}" -eq 4 ]]; then run_row "$slot" "$lane" "$measure" "$tool $version" unavailable "$(tr '\n' ' ' <"$WORK/err.$slot" | cut -c1-500)" diff --git a/plugins/code-metrics/scripts/dispatch.test.sh b/plugins/code-metrics/scripts/dispatch.test.sh index fc7a16d2fc..835caa1da8 100755 --- a/plugins/code-metrics/scripts/dispatch.test.sh +++ b/plugins/code-metrics/scripts/dispatch.test.sh @@ -105,6 +105,8 @@ assert_doc "status empty and every row unavailable with a reason" "$out" \ 'd["status"]=="empty" and d["run"] and all(r["status"]!="ok" and r["reason"] for r in d["run"]) and len(d["unavailable"])==6 and d["measures"]==[]' assert_doc "the reason names both rungs and the install hint" "$out" \ '"scc: scc not on PATH" in d["run"][0]["reason"] and "line-counter: disabled by CODE_METRICS_DISABLE_BUNDLED" in d["run"][0]["reason"] and "boyter/scc" in d["run"][0]["reason"]' +assert_doc "an unavailable row also carries the first install hint as its own field" "$out" \ + 'd["run"][0]["hint"] and "boyter/scc" in d["run"][0]["hint"]' # 4. A ladder row whose adapter does not exist is reported, not skipped. ladder="$(mktemp)" @@ -139,7 +141,7 @@ assert_eq "missing skill name exits 2" 2 "$?" empty_dir="$(mktemp -d)" out="$(PATH="$EMPTY_PATH" bash "$SCRIPT" audit-size --measures file_lines --all "$empty_dir")" assert_doc "empty scope yields one not-applicable row" "$out" \ - 'd["status"]=="empty" and d["scope"]["files"]==0 and d["run"]==[{"lane":"*","measure":"*","collector":None,"status":"not-applicable","reason":"no measurable files in scope"}]' + 'd["status"]=="empty" and d["scope"]["files"]==0 and d["run"]==[{"lane":"*","measure":"*","collector":None,"status":"not-applicable","reason":"no measurable files in scope","hint":None}]' rmdir "$empty_dir" # 8. A collector that probes but fails in collect: exit 3, row unavailable. @@ -484,6 +486,28 @@ case "$err" in esac rm -rf "$repo" "$home" +# 25. A collector that leaves inputs out reports the lane as partial with its +# reason, through the partial-reason file the dispatcher hands every +# collect. The jscpd adapter skips both cluster copies under a 100-byte +# cap, so the fake `jscpd` (probe only; never reached for collect) has +# nothing to replay. +skipper="$(mktemp -d)" +cat >"$skipper/jscpd" <<'EOF' +#!/usr/bin/env bash +if [[ "${1:-}" == "--version" ]]; then printf 'jscpd 5.2.0\n'; exit 0; fi +printf 'jscpd should not have been invoked\n' >&2 +exit 1 +EOF +chmod +x "$skipper/jscpd" +out="$(PATH="$skipper:$EMPTY_PATH" CODE_METRICS_DUP_MAX_SIZE=100 bash "$SCRIPT" audit-duplication --measures duplication --all "$SOURCES/cluster")" +rc=$? +assert_eq "a run whose inputs were all skipped exits 0" 0 "$rc" +assert_doc "a skipped input makes the lane row partial with the reason" "$out" \ + 'any(r["lane"]=="bash" and r["measure"]=="duplication" and r["status"]=="partial" and r["reason"].startswith("2 of 2 files skipped by duplication.max_size 100") for r in d["run"])' +assert_doc "a skipped input leaves the row's hint null" "$out" \ + 'all(r.get("hint") is None for r in d["run"])' +rm -rf "$skipper" + # 20. The default scope exclusions drop dependency and build-output directories # and the scope names each pattern's count. repo="$(mktemp -d)" diff --git a/plugins/code-metrics/scripts/fixtures/clone-classes/aligned/a/shared/shared-utils.sh b/plugins/code-metrics/scripts/fixtures/clone-classes/aligned/a/shared/shared-utils.sh new file mode 100644 index 0000000000..027c163dad --- /dev/null +++ b/plugins/code-metrics/scripts/fixtures/clone-classes/aligned/a/shared/shared-utils.sh @@ -0,0 +1,41 @@ +# shellcheck shell=bash +# Fixture source for the code-metrics duplication suites: a helper vendored +# byte-identical into two sibling plugin directories, standing in for a +# repository that deliberately replicates one path across its plugins. Never +# executed, so it carries no shebang and no exec bit; kept lint-clean on +# purpose. The copy under the sibling directory is byte-for-byte this file. + +HARVEST_LABEL="harvest" + +announce_start() { + local subject="$1" + printf 'start %s %s\n' "$HARVEST_LABEL" "$subject" +} + +announce_finish() { + local subject="$1" + local outcome="${2:-unknown}" + printf 'finish %s %s %s\n' "$HARVEST_LABEL" "$subject" "$outcome" +} + +collect_orchard() { + local basket="$1" + shift + local apple + for apple in "$@"; do + if [[ -z "$apple" ]]; then + continue + fi + printf '%s/%s\n' "$basket" "$apple" + done +} + +measure_basket() { + local basket="$1" + if [[ -d "$basket" ]]; then + find "$basket" -type f | wc -l + return 0 + fi + printf '0\n' + return 1 +} diff --git a/plugins/code-metrics/scripts/fixtures/clone-classes/aligned/b/shared/shared-utils.sh b/plugins/code-metrics/scripts/fixtures/clone-classes/aligned/b/shared/shared-utils.sh new file mode 100644 index 0000000000..027c163dad --- /dev/null +++ b/plugins/code-metrics/scripts/fixtures/clone-classes/aligned/b/shared/shared-utils.sh @@ -0,0 +1,41 @@ +# shellcheck shell=bash +# Fixture source for the code-metrics duplication suites: a helper vendored +# byte-identical into two sibling plugin directories, standing in for a +# repository that deliberately replicates one path across its plugins. Never +# executed, so it carries no shebang and no exec bit; kept lint-clean on +# purpose. The copy under the sibling directory is byte-for-byte this file. + +HARVEST_LABEL="harvest" + +announce_start() { + local subject="$1" + printf 'start %s %s\n' "$HARVEST_LABEL" "$subject" +} + +announce_finish() { + local subject="$1" + local outcome="${2:-unknown}" + printf 'finish %s %s %s\n' "$HARVEST_LABEL" "$subject" "$outcome" +} + +collect_orchard() { + local basket="$1" + shift + local apple + for apple in "$@"; do + if [[ -z "$apple" ]]; then + continue + fi + printf '%s/%s\n' "$basket" "$apple" + done +} + +measure_basket() { + local basket="$1" + if [[ -d "$basket" ]]; then + find "$basket" -type f | wc -l + return 0 + fi + printf '0\n' + return 1 +} diff --git a/plugins/code-metrics/scripts/fixtures/clone-classes/aligned/c/shared/shared-utils.sh b/plugins/code-metrics/scripts/fixtures/clone-classes/aligned/c/shared/shared-utils.sh new file mode 100644 index 0000000000..027c163dad --- /dev/null +++ b/plugins/code-metrics/scripts/fixtures/clone-classes/aligned/c/shared/shared-utils.sh @@ -0,0 +1,41 @@ +# shellcheck shell=bash +# Fixture source for the code-metrics duplication suites: a helper vendored +# byte-identical into two sibling plugin directories, standing in for a +# repository that deliberately replicates one path across its plugins. Never +# executed, so it carries no shebang and no exec bit; kept lint-clean on +# purpose. The copy under the sibling directory is byte-for-byte this file. + +HARVEST_LABEL="harvest" + +announce_start() { + local subject="$1" + printf 'start %s %s\n' "$HARVEST_LABEL" "$subject" +} + +announce_finish() { + local subject="$1" + local outcome="${2:-unknown}" + printf 'finish %s %s %s\n' "$HARVEST_LABEL" "$subject" "$outcome" +} + +collect_orchard() { + local basket="$1" + shift + local apple + for apple in "$@"; do + if [[ -z "$apple" ]]; then + continue + fi + printf '%s/%s\n' "$basket" "$apple" + done +} + +measure_basket() { + local basket="$1" + if [[ -d "$basket" ]]; then + find "$basket" -type f | wc -l + return 0 + fi + printf '0\n' + return 1 +} diff --git a/plugins/code-metrics/scripts/fixtures/clone-classes/offset/c1.sh b/plugins/code-metrics/scripts/fixtures/clone-classes/offset/c1.sh new file mode 100644 index 0000000000..27b4041c3d --- /dev/null +++ b/plugins/code-metrics/scripts/fixtures/clone-classes/offset/c1.sh @@ -0,0 +1,38 @@ +# shellcheck shell=bash +# Offset fixture: the shared fragment begins on line 3 of this copy. +HARVEST_LABEL="harvest" + +announce_start() { + local subject="$1" + printf 'start %s %s\n' "$HARVEST_LABEL" "$subject" +} + +announce_finish() { + local subject="$1" + local outcome="${2:-unknown}" + printf 'finish %s %s %s\n' "$HARVEST_LABEL" "$subject" "$outcome" +} + +collect_orchard() { + local basket="$1" + shift + local apple + for apple in "$@"; do + if [[ -z "$apple" ]]; then + continue + fi + printf '%s/%s\n' "$basket" "$apple" + done +} + +measure_basket() { + local basket="$1" + if [[ -d "$basket" ]]; then + find "$basket" -type f | wc -l + return 0 + fi + printf '0\n' + return 1 +} + +announce_start "offset-one" diff --git a/plugins/code-metrics/scripts/fixtures/clone-classes/offset/c2.sh b/plugins/code-metrics/scripts/fixtures/clone-classes/offset/c2.sh new file mode 100644 index 0000000000..176809c895 --- /dev/null +++ b/plugins/code-metrics/scripts/fixtures/clone-classes/offset/c2.sh @@ -0,0 +1,39 @@ +# shellcheck shell=bash +# Offset fixture: the shared fragment begins on line 4 of this copy, one +# line lower than in the first copy, so no instance range aligns with it. +HARVEST_LABEL="harvest" + +announce_start() { + local subject="$1" + printf 'start %s %s\n' "$HARVEST_LABEL" "$subject" +} + +announce_finish() { + local subject="$1" + local outcome="${2:-unknown}" + printf 'finish %s %s %s\n' "$HARVEST_LABEL" "$subject" "$outcome" +} + +collect_orchard() { + local basket="$1" + shift + local apple + for apple in "$@"; do + if [[ -z "$apple" ]]; then + continue + fi + printf '%s/%s\n' "$basket" "$apple" + done +} + +measure_basket() { + local basket="$1" + if [[ -d "$basket" ]]; then + find "$basket" -type f | wc -l + return 0 + fi + printf '0\n' + return 1 +} + +announce_finish "offset-two" "done" diff --git a/plugins/code-metrics/scripts/fixtures/clone-classes/offset/c3.sh b/plugins/code-metrics/scripts/fixtures/clone-classes/offset/c3.sh new file mode 100644 index 0000000000..f4ff674ba4 --- /dev/null +++ b/plugins/code-metrics/scripts/fixtures/clone-classes/offset/c3.sh @@ -0,0 +1,39 @@ +# shellcheck shell=bash +# Offset fixture: this copy carries only the first part of the shared +# fragment, beginning on line 5, so the range it shares with the first copy +# is shorter than the range the first and second copies share. The two pairs +# jscpd reports therefore name the first copy with two different ranges and +# stay two clone groups: the merge joins pairs on an identical instance, not +# on an overlapping one. +HARVEST_LABEL="harvest" + +announce_start() { + local subject="$1" + printf 'start %s %s\n' "$HARVEST_LABEL" "$subject" +} + +announce_finish() { + local subject="$1" + local outcome="${2:-unknown}" + printf 'finish %s %s %s\n' "$HARVEST_LABEL" "$subject" "$outcome" +} + +collect_orchard() { + local basket="$1" + shift + local apple + for apple in "$@"; do + if [[ -z "$apple" ]]; then + continue + fi + printf '%s/%s\n' "$basket" "$apple" + done +} + +weigh_basket() { + local basket="$1" + local weight="${2:-0}" + printf 'weigh %s %s\n' "$basket" "$weight" +} + +weigh_basket "offset-three" 7 diff --git a/plugins/code-metrics/scripts/fixtures/registry/cluster.txt b/plugins/code-metrics/scripts/fixtures/registry/cluster.txt index fdcc0ccc5b..5e064b56c8 100644 --- a/plugins/code-metrics/scripts/fixtures/registry/cluster.txt +++ b/plugins/code-metrics/scripts/fixtures/registry/cluster.txt @@ -2,8 +2,14 @@ # the shape a consuming repository declares its own deliberate replication # (this repository's scripts/cross-plugin-source-registry.txt is the live # example): one path-within-plugin per line, `#` comments and blank lines -# ignored. The cluster under fixtures/sources/cluster carries this path -# byte-identical in alpha and beta on purpose, so a clone over the two copies -# is an exclusion rather than duplication debt. +# ignored, or a cluster line ` -> ...` naming a +# root-relative canonical copy and the paths or globs that carry it. The +# cluster under fixtures/sources/cluster carries this path byte-identical in +# alpha and beta on purpose, so a clone over the two copies is an exclusion +# rather than duplication debt. shared/shared-utils.sh + +# The same cluster written as a cluster line, with a root canonical; left as a +# comment because the fixture tree carries no root copy. +# lib/shared-utils.sh -> plugins/*/shared/shared-utils.sh diff --git a/plugins/code-metrics/scripts/fixtures/tool-output/jscpd-aligned3.json b/plugins/code-metrics/scripts/fixtures/tool-output/jscpd-aligned3.json new file mode 100644 index 0000000000..4af8a21c1b --- /dev/null +++ b/plugins/code-metrics/scripts/fixtures/tool-output/jscpd-aligned3.json @@ -0,0 +1,109 @@ +{ + "duplicates": [ + { + "firstFile": { + "end": 41, + "endLoc": { + "column": 1, + "line": 41, + "position": 1002 + }, + "name": "plugins/code-metrics/scripts/fixtures/clone-classes/aligned/a/shared/shared-utils.sh", + "start": 1, + "startLoc": { + "column": 0, + "line": 1, + "position": 0 + } + }, + "format": "bash", + "fragment": "# shellcheck shell=bash\n# Fixture source for the code-metrics duplication suites: a helper vendored\n# byte-identical into two sibling plugin directories, standing in for a\n# repository that deliberately replicates one path across its plugins. Never\n# executed, so it carries no shebang and no exec bit; kept lint-clean on\n# purpose. The copy under the sibling directory is byte-for-byte this file.\n\nHARVEST_LABEL=\"harvest\"\n\nannounce_start() {\n local subject=\"$1\"\n printf 'start %s %s\\n' \"$HARVEST_LABEL\" \"$subject\"\n}\n\nannounce_finish() {\n local subject=\"$1\"\n local outcome=\"${2:-unknown}\"\n printf 'finish %s %s %s\\n' \"$HARVEST_LABEL\" \"$subject\" \"$outcome\"\n}\n\ncollect_orchard() {\n local basket=\"$1\"\n shift\n local apple\n for apple in \"$@\"; do\n if [[ -z \"$apple\" ]]; then\n continue\n fi\n printf '%s/%s\\n' \"$basket\" \"$apple\"\n done\n}\n\nmeasure_basket() {\n local basket=\"$1\"\n if [[ -d \"$basket\" ]]; then\n find \"$basket\" -type f | wc -l\n return 0\n fi\n printf '0\\n'\n return 1\n}", + "isNew": false, + "kind": "exact", + "lines": 41, + "secondFile": { + "end": 41, + "endLoc": { + "column": 1, + "line": 41, + "position": 1002 + }, + "name": "plugins/code-metrics/scripts/fixtures/clone-classes/aligned/b/shared/shared-utils.sh", + "start": 1, + "startLoc": { + "column": 0, + "line": 1, + "position": 0 + } + }, + "tokens": 110 + }, + { + "firstFile": { + "end": 41, + "endLoc": { + "column": 1, + "line": 41, + "position": 1002 + }, + "name": "plugins/code-metrics/scripts/fixtures/clone-classes/aligned/a/shared/shared-utils.sh", + "start": 1, + "startLoc": { + "column": 0, + "line": 1, + "position": 0 + } + }, + "format": "bash", + "fragment": "# shellcheck shell=bash\n# Fixture source for the code-metrics duplication suites: a helper vendored\n# byte-identical into two sibling plugin directories, standing in for a\n# repository that deliberately replicates one path across its plugins. Never\n# executed, so it carries no shebang and no exec bit; kept lint-clean on\n# purpose. The copy under the sibling directory is byte-for-byte this file.\n\nHARVEST_LABEL=\"harvest\"\n\nannounce_start() {\n local subject=\"$1\"\n printf 'start %s %s\\n' \"$HARVEST_LABEL\" \"$subject\"\n}\n\nannounce_finish() {\n local subject=\"$1\"\n local outcome=\"${2:-unknown}\"\n printf 'finish %s %s %s\\n' \"$HARVEST_LABEL\" \"$subject\" \"$outcome\"\n}\n\ncollect_orchard() {\n local basket=\"$1\"\n shift\n local apple\n for apple in \"$@\"; do\n if [[ -z \"$apple\" ]]; then\n continue\n fi\n printf '%s/%s\\n' \"$basket\" \"$apple\"\n done\n}\n\nmeasure_basket() {\n local basket=\"$1\"\n if [[ -d \"$basket\" ]]; then\n find \"$basket\" -type f | wc -l\n return 0\n fi\n printf '0\\n'\n return 1\n}", + "isNew": false, + "kind": "exact", + "lines": 41, + "secondFile": { + "end": 41, + "endLoc": { + "column": 1, + "line": 41, + "position": 1002 + }, + "name": "plugins/code-metrics/scripts/fixtures/clone-classes/aligned/c/shared/shared-utils.sh", + "start": 1, + "startLoc": { + "column": 0, + "line": 1, + "position": 0 + } + }, + "tokens": 110 + } + ], + "statistics": { + "detectionDate": "2026-09-11T16:00:18.373Z", + "formats": { + "bash": { + "clones": 2, + "duplicatedLines": 80, + "duplicatedTokens": 220, + "lines": 123, + "newClones": 0, + "newDuplicatedLines": 0, + "percentage": 65.04065040650406, + "percentageTokens": 66.66666666666666, + "sources": 3, + "tokens": 330 + } + }, + "total": { + "clones": 2, + "duplicatedLines": 80, + "duplicatedTokens": 220, + "lines": 123, + "newClones": 0, + "newDuplicatedLines": 0, + "percentage": 65.04065040650406, + "percentageTokens": 66.66666666666666, + "sources": 3, + "tokens": 330 + } + } +} diff --git a/plugins/code-metrics/scripts/fixtures/tool-output/jscpd-offset3.json b/plugins/code-metrics/scripts/fixtures/tool-output/jscpd-offset3.json new file mode 100644 index 0000000000..aae97fbe47 --- /dev/null +++ b/plugins/code-metrics/scripts/fixtures/tool-output/jscpd-offset3.json @@ -0,0 +1,109 @@ +{ + "duplicates": [ + { + "firstFile": { + "end": 36, + "endLoc": { + "column": 1, + "line": 36, + "position": 696 + }, + "name": "plugins/code-metrics/scripts/fixtures/clone-classes/offset/c1.sh", + "start": 3, + "startLoc": { + "column": 0, + "line": 3, + "position": 93 + } + }, + "format": "bash", + "fragment": "HARVEST_LABEL=\"harvest\"\n\nannounce_start() {\n local subject=\"$1\"\n printf 'start %s %s\\n' \"$HARVEST_LABEL\" \"$subject\"\n}\n\nannounce_finish() {\n local subject=\"$1\"\n local outcome=\"${2:-unknown}\"\n printf 'finish %s %s %s\\n' \"$HARVEST_LABEL\" \"$subject\" \"$outcome\"\n}\n\ncollect_orchard() {\n local basket=\"$1\"\n shift\n local apple\n for apple in \"$@\"; do\n if [[ -z \"$apple\" ]]; then\n continue\n fi\n printf '%s/%s\\n' \"$basket\" \"$apple\"\n done\n}\n\nmeasure_basket() {\n local basket=\"$1\"\n if [[ -d \"$basket\" ]]; then\n find \"$basket\" -type f | wc -l\n return 0\n fi\n printf '0\\n'\n return 1\n}", + "isNew": false, + "kind": "exact", + "lines": 34, + "secondFile": { + "end": 37, + "endLoc": { + "column": 1, + "line": 37, + "position": 774 + }, + "name": "plugins/code-metrics/scripts/fixtures/clone-classes/offset/c2.sh", + "start": 4, + "startLoc": { + "column": 0, + "line": 4, + "position": 171 + } + }, + "tokens": 104 + }, + { + "firstFile": { + "end": 26, + "endLoc": { + "column": 1, + "line": 26, + "position": 545 + }, + "name": "plugins/code-metrics/scripts/fixtures/clone-classes/offset/c1.sh", + "start": 3, + "startLoc": { + "column": 0, + "line": 3, + "position": 93 + } + }, + "format": "bash", + "fragment": "HARVEST_LABEL=\"harvest\"\n\nannounce_start() {\n local subject=\"$1\"\n printf 'start %s %s\\n' \"$HARVEST_LABEL\" \"$subject\"\n}\n\nannounce_finish() {\n local subject=\"$1\"\n local outcome=\"${2:-unknown}\"\n printf 'finish %s %s %s\\n' \"$HARVEST_LABEL\" \"$subject\" \"$outcome\"\n}\n\ncollect_orchard() {\n local basket=\"$1\"\n shift\n local apple\n for apple in \"$@\"; do\n if [[ -z \"$apple\" ]]; then\n continue\n fi\n printf '%s/%s\\n' \"$basket\" \"$apple\"\n done\n}", + "isNew": false, + "kind": "exact", + "lines": 24, + "secondFile": { + "end": 31, + "endLoc": { + "column": 1, + "line": 31, + "position": 877 + }, + "name": "plugins/code-metrics/scripts/fixtures/clone-classes/offset/c3.sh", + "start": 8, + "startLoc": { + "column": 0, + "line": 8, + "position": 425 + } + }, + "tokens": 69 + } + ], + "statistics": { + "detectionDate": "2026-09-11T16:02:53.170Z", + "formats": { + "bash": { + "clones": 2, + "duplicatedLines": 56, + "duplicatedTokens": 173, + "lines": 116, + "newClones": 0, + "newDuplicatedLines": 0, + "percentage": 48.275862068965516, + "percentageTokens": 55.095541401273884, + "sources": 3, + "tokens": 314 + } + }, + "total": { + "clones": 2, + "duplicatedLines": 56, + "duplicatedTokens": 173, + "lines": 116, + "newClones": 0, + "newDuplicatedLines": 0, + "percentage": 48.275862068965516, + "percentageTokens": 55.095541401273884, + "sources": 3, + "tokens": 314 + } + } +} diff --git a/plugins/code-metrics/scripts/fixtures/tool-output/jscpd.json b/plugins/code-metrics/scripts/fixtures/tool-output/jscpd.json index 3cc07f78b3..f1476b5721 100644 --- a/plugins/code-metrics/scripts/fixtures/tool-output/jscpd.json +++ b/plugins/code-metrics/scripts/fixtures/tool-output/jscpd.json @@ -1,70 +1,71 @@ { - "duplicates": [ - { - "firstFile": { - "end": 41, - "endLoc": { - "column": 1, - "line": 41, - "position": 1002 - }, - "name": "plugins/code-metrics/scripts/fixtures/sources/cluster/alpha/shared/shared-utils.sh", - "start": 1, - "startLoc": { - "column": 0, - "line": 1, - "position": 0 - } - }, - "format": "bash", - "fragment": "# shellcheck shell=bash\n# Fixture source for the code-metrics duplication suites: a helper vendored\n# byte-identical into two sibling plugin directories, standing in for a\n# repository that deliberately replicates one path across its plugins. Never\n# executed, so it carries no shebang and no exec bit; kept lint-clean on\n# purpose. The copy under the sibling directory is byte-for-byte this file.\n\nHARVEST_LABEL=\"harvest\"\n\nannounce_start() {\n local subject=\"$1\"\n printf 'start %s %s\\n' \"$HARVEST_LABEL\" \"$subject\"\n}\n\nannounce_finish() {\n local subject=\"$1\"\n local outcome=\"${2:-unknown}\"\n printf 'finish %s %s %s\\n' \"$HARVEST_LABEL\" \"$subject\" \"$outcome\"\n}\n\ncollect_orchard() {\n local basket=\"$1\"\n shift\n local apple\n for apple in \"$@\"; do\n if [[ -z \"$apple\" ]]; then\n continue\n fi\n printf '%s/%s\\n' \"$basket\" \"$apple\"\n done\n}\n\nmeasure_basket() {\n local basket=\"$1\"\n if [[ -d \"$basket\" ]]; then\n find \"$basket\" -type f | wc -l\n return 0\n fi\n printf '0\\n'\n return 1\n}", - "isNew": false, - "lines": 41, - "secondFile": { - "end": 41, - "endLoc": { - "column": 1, - "line": 41, - "position": 1002 - }, - "name": "plugins/code-metrics/scripts/fixtures/sources/cluster/beta/shared/shared-utils.sh", - "start": 1, - "startLoc": { - "column": 0, - "line": 1, - "position": 0 - } - }, - "tokens": 110 + "duplicates": [ + { + "firstFile": { + "end": 41, + "endLoc": { + "column": 1, + "line": 41, + "position": 1002 + }, + "name": "plugins/code-metrics/scripts/fixtures/sources/cluster/alpha/shared/shared-utils.sh", + "start": 1, + "startLoc": { + "column": 0, + "line": 1, + "position": 0 } - ], - "statistics": { - "detectionDate": "2026-09-05T16:06:08.332Z", - "formats": { - "bash": { - "clones": 1, - "duplicatedLines": 40, - "duplicatedTokens": 110, - "lines": 82, - "newClones": 0, - "newDuplicatedLines": 0, - "percentage": 48.78048780487805, - "percentageTokens": 50.0, - "sources": 2, - "tokens": 220 - } + }, + "format": "bash", + "fragment": "# shellcheck shell=bash\n# Fixture source for the code-metrics duplication suites: a helper vendored\n# byte-identical into two sibling plugin directories, standing in for a\n# repository that deliberately replicates one path across its plugins. Never\n# executed, so it carries no shebang and no exec bit; kept lint-clean on\n# purpose. The copy under the sibling directory is byte-for-byte this file.\n\nHARVEST_LABEL=\"harvest\"\n\nannounce_start() {\n local subject=\"$1\"\n printf 'start %s %s\\n' \"$HARVEST_LABEL\" \"$subject\"\n}\n\nannounce_finish() {\n local subject=\"$1\"\n local outcome=\"${2:-unknown}\"\n printf 'finish %s %s %s\\n' \"$HARVEST_LABEL\" \"$subject\" \"$outcome\"\n}\n\ncollect_orchard() {\n local basket=\"$1\"\n shift\n local apple\n for apple in \"$@\"; do\n if [[ -z \"$apple\" ]]; then\n continue\n fi\n printf '%s/%s\\n' \"$basket\" \"$apple\"\n done\n}\n\nmeasure_basket() {\n local basket=\"$1\"\n if [[ -d \"$basket\" ]]; then\n find \"$basket\" -type f | wc -l\n return 0\n fi\n printf '0\\n'\n return 1\n}", + "isNew": false, + "kind": "exact", + "lines": 41, + "secondFile": { + "end": 41, + "endLoc": { + "column": 1, + "line": 41, + "position": 1002 }, - "total": { - "clones": 1, - "duplicatedLines": 40, - "duplicatedTokens": 110, - "lines": 82, - "newClones": 0, - "newDuplicatedLines": 0, - "percentage": 48.78048780487805, - "percentageTokens": 50.0, - "sources": 2, - "tokens": 220 + "name": "plugins/code-metrics/scripts/fixtures/sources/cluster/beta/shared/shared-utils.sh", + "start": 1, + "startLoc": { + "column": 0, + "line": 1, + "position": 0 } + }, + "tokens": 110 + } + ], + "statistics": { + "detectionDate": "2026-09-11T16:41:08.069Z", + "formats": { + "bash": { + "clones": 1, + "duplicatedLines": 40, + "duplicatedTokens": 110, + "lines": 82, + "newClones": 0, + "newDuplicatedLines": 0, + "percentage": 48.78048780487805, + "percentageTokens": 50.0, + "sources": 2, + "tokens": 220 + } + }, + "total": { + "clones": 1, + "duplicatedLines": 40, + "duplicatedTokens": 110, + "lines": 82, + "newClones": 0, + "newDuplicatedLines": 0, + "percentage": 48.78048780487805, + "percentageTokens": 50.0, + "sources": 2, + "tokens": 220 } + } } diff --git a/plugins/code-metrics/scripts/pathglob.py b/plugins/code-metrics/scripts/pathglob.py index 47a1e91137..b62069024a 100755 --- a/plugins/code-metrics/scripts/pathglob.py +++ b/plugins/code-metrics/scripts/pathglob.py @@ -25,6 +25,7 @@ from __future__ import annotations +import os import re import sys @@ -40,6 +41,27 @@ def _normalize(path: str) -> str: return path +def root_relative(path: str, root: str) -> str: + """The path relative to `root` with forward slashes; unchanged without a root. + + A cwd-relative path is joined onto the working directory first, so a run + from a subdirectory (where the dispatcher names files `../../lib/x.sh`) and + a run from the root name a file the same way. Shared by the report + summarizer, the registry filter, and the clone-class merge so the three + never disagree on what "root-relative" means. + """ + path = (path or "").replace("\\", "/") + if root: + absolute = path if os.path.isabs(path) else os.path.join(os.getcwd(), path) + try: + path = os.path.relpath(absolute, root).replace("\\", "/") + except ValueError: + pass + while path.startswith("./"): + path = path[2:] + return path + + def translate(pattern: str) -> str: """Return an anchored regex for a gitignore-style glob.""" pattern = _normalize(pattern) diff --git a/plugins/code-metrics/scripts/replica-collapse.py b/plugins/code-metrics/scripts/replica-collapse.py index 7bb680858f..2c4df29ec7 100755 --- a/plugins/code-metrics/scripts/replica-collapse.py +++ b/plugins/code-metrics/scripts/replica-collapse.py @@ -52,6 +52,12 @@ def read_registry(path: str) -> list[tuple[int, str]]: line = raw.strip() if not line or line.startswith("#"): continue + # A cluster line (` -> ...`) names a root + # canonical outside any plugin and the globs that carry it; it is + # the clone-group reader's (registry-filter.py) and never a + # path-within-plugin, so this pass leaves it alone. + if " -> " in line: + continue entries.append((number, line.replace("\\", "/").lstrip("/"))) return entries @@ -170,7 +176,10 @@ def main(argv: list[str]) -> int: try: document = json.load(sys.stdin) except (json.JSONDecodeError, ValueError) as exc: - print(f"replica-collapse.py: stdin is not a JSON document ({exc})", file=sys.stderr) + print( + f"replica-collapse.py: stdin is not a JSON document ({exc})", + file=sys.stderr, + ) return 2 print(json.dumps(collapse(document, registries, args.prefix), indent=2)) return 0 @@ -178,6 +187,9 @@ def main(argv: list[str]) -> int: if __name__ == "__main__": if sys.version_info < MIN_PYTHON: - print("replica-collapse.py needs Python %d.%d or later" % MIN_PYTHON, file=sys.stderr) + print( + "replica-collapse.py needs Python %d.%d or later" % MIN_PYTHON, + file=sys.stderr, + ) sys.exit(2) sys.exit(main(sys.argv[1:])) diff --git a/plugins/code-metrics/scripts/report.py b/plugins/code-metrics/scripts/report.py index 26ac4cc1b9..26159cc46b 100755 --- a/plugins/code-metrics/scripts/report.py +++ b/plugins/code-metrics/scripts/report.py @@ -10,25 +10,32 @@ report.py assemble --skill --scope --run --measures --thresholds - [--excluded ] + [--excluded ] [--root ] Print the report document: `run[]` is the coverage-of-this-run table, `measures[]` gains `over_reference`, `summary` counts, `unavailable[]` lists every non-ok lane/measure, and `status` is complete, partial, or - empty. A value that was not measured is `null`, never zero. + empty. A value that was not measured is `null`, never zero. A `partial` + run row counts as measured, so a lane that skipped every file is + `partial`, not `empty`. - report.py render [--document ] [< report.json] + report.py render [--document ] [--rollup-depth ] [< report.json] Print the markdown rendering of a report document read from stdin. The table joins the rows every collector produced for one function into one line (the JSON keeps one row per collector), lists rows over a reference first by how far past it they sit, and caps itself at MAX_RENDERED_ROWS; `--document` names the file the caller persisted the whole document to, - so the cap line and the summary can point at it. + so the cap line and the summary can point at it. A duplication document + (clone-group rows, or `skill` audit-duplication) lists groups largest + first, adds a `## Rollup` section with per-lane and per-directory tables + (directories to `--rollup-depth`, default 2), and summarizes as + `Files with clones`; every other document renders as it always has. - report.py resummarize [< report.json] + report.py resummarize [--root ] [< report.json] Recompute `summary` from `measures[]` and print the document; for a skill that drops rows after assembly (a duplication registry moving clone groups into `excluded[]`). Clone-group rows (`instances[]`) add - `summary.duplicated_lines` and `summary.clone_groups`. + `summary.duplicated_lines`, `summary.clone_groups`, `summary.by_lane`, + and `summary.by_directory` (paths made relative to `--root`). Exit 0 on success, 2 on a usage error or unreadable input. """ @@ -41,6 +48,8 @@ import sys from typing import Any +from pathglob import root_relative + MIN_PYTHON = (3, 9) SCHEMA = "code-metrics/v1" RUN_STATUSES = ("ok", "partial", "unavailable", "not-applicable", "deferred") @@ -112,14 +121,35 @@ def _over(threshold: dict[str, Any], value: Any) -> bool: return value >= reference -def summarize(measures: list[dict[str, Any]]) -> dict[str, Any]: +def _ancestors(path: str) -> list[str]: + """`.` and every directory above the file, root first.""" + parts = path.split("/")[:-1] + return ["."] + ["/".join(parts[: index + 1]) for index in range(len(parts))] + + +def _tally(buckets: dict[str, dict[str, int]], key: str, lines: int) -> None: + bucket = buckets.setdefault(key, {"groups": 0, "duplicated_lines": 0}) + bucket["groups"] += 1 + bucket["duplicated_lines"] += lines + + +def summarize(measures: list[dict[str, Any]], root: str = "") -> dict[str, Any]: """The `summary` block, derived from `measures[]` alone so a skill that drops rows after assembly (a duplication registry exclusion) can recompute it through the `resummarize` verb. Counts use each row's `over_reference` list as assembled; clone-group rows (those carrying `instances[]`) add `duplicated_lines` (sum of `values.lines`, each group counted once) and - `clone_groups`, and their instance files count toward `files`.""" + `clone_groups`, and their instance files count toward `files`. + + Clone-group rows also add `by_lane` (lane to `{groups, duplicated_lines}`) + and `by_directory` (the same shape for `.` and every ancestor directory of + each group's first instance, made relative to `root`). A group counts once + per ancestor, so a parent includes its children and the rows cannot be + summed, while `by_directory["."]` and the per-lane sum both restate the + totals.""" files: set[str] = set() + by_lane: dict[str, dict[str, int]] = {} + by_directory: dict[str, dict[str, int]] = {} # (file, name) -> the distinct start lines reported for it. A name is not an # identity: one file can hold two `render` methods. A start line is not one # either, because a collector that reports Halstead for a function need not @@ -156,11 +186,17 @@ def summarize(measures: list[dict[str, Any]]) -> dict[str, Any]: if instances: clone_groups += 1 lines = (row.get("values") or {}).get("lines") + counted = 0 if isinstance(lines, (int, float)) and not isinstance(lines, bool): - duplicated_lines += int(lines) + counted = int(lines) + duplicated_lines += counted for instance in instances: if instance.get("file"): files.add(instance["file"]) + _tally(by_lane, str(row.get("lane") or "*"), counted) + first = root_relative(str(instances[0].get("file") or ""), root) + for directory in _ancestors(first): + _tally(by_directory, directory, counted) summary: dict[str, Any] = { "files": len(files), "functions": sum(max(1, len(starts)) for starts in functions.values()), @@ -169,6 +205,8 @@ def summarize(measures: list[dict[str, Any]]) -> dict[str, Any]: if clone_groups: summary["duplicated_lines"] = duplicated_lines summary["clone_groups"] = clone_groups + summary["by_lane"] = by_lane + summary["by_directory"] = by_directory return summary @@ -179,6 +217,7 @@ def assemble( measures: list[dict[str, Any]], threshold_entries: list[dict[str, Any]], excluded: list[dict[str, Any]], + root: str = "", ) -> dict[str, Any]: for row in run: if row.get("status") not in RUN_STATUSES: @@ -196,10 +235,13 @@ def assemble( # exist for that lane), so it never withholds `complete`; `unavailable`, # `deferred` and `partial` rows do, because something implied was not # measured. `partial` still counts as having produced rows, so a run that - # measured part of a lane reads as `partial` rather than as `empty`. + # measured part of a lane reads as `partial` rather than as `empty`, even + # when it skipped every file and has no row to show: the skip is stated in + # the run row, and "Measured nothing" would contradict it. ok_rows = [row for row in run if row.get("status") in ("ok", "partial")] + partial_rows = [row for row in run if row.get("status") == "partial"] settled = [row for row in run if row.get("status") in ("ok", "not-applicable")] - if not ok_rows or not measures: + if not ok_rows or (not measures and not partial_rows): status = "empty" elif len(settled) == len(run): status = "complete" @@ -219,7 +261,7 @@ def assemble( # documents needs to know which value the reference was applied to. "thresholds": list(threshold_entries), "measures": measures, - "summary": summarize(measures), + "summary": summarize(measures, root), "excluded": excluded, "unavailable": [ f"{row.get('lane', '*')}/{row.get('measure', '*')}" @@ -261,6 +303,30 @@ def _fmt(value: Any) -> str: return str(value) +def _is_duplication(doc: dict[str, Any]) -> bool: + return doc.get("skill") == "audit-duplication" or any( + row.get("instances") for row in doc.get("measures", []) + ) + + +def _depth(directory: str) -> int: + return 0 if directory == "." else directory.count("/") + 1 + + +def _clone_sort_key(row: dict[str, Any]) -> tuple[int, int, str]: + values = row.get("values") or {} + instances = row.get("instances") or [{}] + + def number(value: Any) -> int: + return int(value) if isinstance(value, (int, float)) else 0 + + return ( + -number(values.get("lines")), + -number(values.get("tokens")), + str(instances[0].get("file") or ""), + ) + + def _is_number(value: Any) -> bool: return isinstance(value, (int, float)) and not isinstance(value, bool) @@ -375,13 +441,37 @@ def _over_distance(row: dict[str, Any], references: dict[str, Any]) -> float: return worst -def render(doc: dict[str, Any], document_path: str | None = None) -> str: +def render( + doc: dict[str, Any], document_path: str | None = None, rollup_depth: int = 2 +) -> str: lines: list[str] = [] status = doc.get("status", "empty") headline = "Measured nothing" if status == "empty" else f"Status: {status}" scope = doc.get("scope", {}) + duplication = _is_duplication(doc) lines.append(f"# code-metrics: {doc.get('skill', '?')}") lines.append("") + # A `not-applicable` row (the `other` lane, which no detector covers) is + # not a probe that failed, so it neither earns the headline nor blocks it. + detector_rows = [ + row + for row in doc.get("run", []) + if row.get("measure") == "duplication" and row.get("status") != "not-applicable" + ] + if ( + duplication + and detector_rows + and all(row.get("status") == "unavailable" for row in detector_rows) + ): + # One headline for the whole run: the lane rows below still carry + # each probe's own reason, so this names the fix once, not per lane. + hint = next((row.get("hint") for row in detector_rows if row.get("hint")), "") + lines.append( + "No clone detector ran in any lane" + + (f": {hint}" if hint else "") + + ". Run `/code-metrics:setup` to install one." + ) + lines.append("") lines.append( f"{headline}. Scope: {scope.get('mode', '?')}" + (f" against `{scope['base']}`" if scope.get("base") else "") @@ -447,25 +537,32 @@ def render(doc: dict[str, Any], document_path: str | None = None) -> str: lines.append("|" + "---|" * (5 + len(keys))) shown = 0 primary = _primary_threshold(thresholds_, keys) - # Rows over a reference come first, the furthest past it at the top, - # so the table's opening lines are the ones a reader came for; the - # rest follow by the primary measure's value, largest first, so a - # size report reads longest to shortest rather than alphabetically. - for row in sorted( - measures, - key=lambda r: ( - # A lane row (`lane-total`) is the lane's figure: it leads its - # table and never falls under the row cap. - 0 if "lane-total" in (r.get("labels") or []) else 1, - -len(r.get("over_reference", [])), - -_over_distance(r, references), - _primary_rank(primary, r), - # A lane row (type debt) has `file: null`; `or ""` keeps it - # comparable with the file rows it now sorts among. - r.get("file") or "", - r.get("start_line") or 0, - ), - ): + if duplication: + # Largest group first: the reader's question is "what is the + # biggest copy", not which file sorts first. + ordered = sorted(measures, key=_clone_sort_key) + else: + # Rows over a reference come first, the furthest past it at the + # top, so the table's opening lines are the ones a reader came + # for; the rest follow by the primary measure's value, largest + # first, so a size report reads longest to shortest rather than + # alphabetically. + ordered = sorted( + measures, + key=lambda r: ( + # A lane row (`lane-total`) is the lane's figure: it leads + # its table and never falls under the row cap. + 0 if "lane-total" in (r.get("labels") or []) else 1, + -len(r.get("over_reference", [])), + -_over_distance(r, references), + _primary_rank(primary, r), + # A lane row (type debt) has `file: null`; `or ""` keeps + # it comparable with the file rows it now sorts among. + r.get("file") or "", + r.get("start_line") or 0, + ), + ) + for row in ordered: if shown >= MAX_RENDERED_ROWS: remaining = len(measures) - shown where_full = ( @@ -509,21 +606,59 @@ def render(doc: dict[str, Any], document_path: str | None = None) -> str: "found no operators or operands in that function." ) summary = doc.get("summary", {}) + by_lane = summary.get("by_lane") or {} + by_directory = summary.get("by_directory") or {} + if duplication and (by_lane or by_directory): + lines.append("") + lines.append("## Rollup") + lines.append("") + lines.append("| Lane | Clone groups | Duplicated lines |") + lines.append("|---|---|---|") + for lane, bucket in sorted(by_lane.items()): + lines.append( + f"| {lane} | {bucket.get('groups', 0)} | {bucket.get('duplicated_lines', 0)} |" + ) + lines.append("") + lines.append( + f"| Directory (to depth {rollup_depth}) | Clone groups | Duplicated lines |" + ) + lines.append("|---|---|---|") + for directory, bucket in sorted(by_directory.items()): + if _depth(directory) <= rollup_depth: + lines.append( + f"| {directory} | {bucket.get('groups', 0)} | " + f"{bucket.get('duplicated_lines', 0)} |" + ) + lines.append("") + lines.append( + "A group is attributed to every directory above its first instance, so a parent " + "includes its children and the directory rows cannot be summed; `.` restates the " + "totals. The JSON carries every directory." + ) lines.append("") lines.append("## Summary") lines.append("") - lines.append( - f"Files: {summary.get('files', 0)}. " - # A file-level report has no functions to count; the figure is printed - # only where function rows exist. - + (f"Functions: {summary['functions']}. " if summary.get("functions") else "") - + "Over reference: " - + ( - ", ".join(f"{k} {v}" for k, v in summary.get("over_reference", {}).items()) - or "none" + if duplication: + lines.append(f"Files with clones: {summary.get('files', 0)}.") + else: + lines.append( + f"Files: {summary.get('files', 0)}. " + # A file-level report has no functions to count; the figure is + # printed only where function rows exist. + + ( + f"Functions: {summary['functions']}. " + if summary.get("functions") + else "" + ) + + "Over reference: " + + ( + ", ".join( + f"{k} {v}" for k, v in summary.get("over_reference", {}).items() + ) + or "none" + ) + + "." ) - + "." - ) if "duplicated_lines" in summary: lines.append( f"Duplicated lines: {summary['duplicated_lines']} in " @@ -533,6 +668,11 @@ def render(doc: dict[str, Any], document_path: str | None = None) -> str: lines.append( f"Excluded by a sanctioned-replication registry: {len(doc['excluded'])}." ) + elif duplication: + lines.append( + "Excluded by a sanctioned-replication registry: 0 (no registry configured, or " + "none matched)." + ) replicated_rows = [r for r in measures if (r.get("replicas") or {}).get("count")] if replicated_rows: standing_for = sum(int(r["replicas"]["count"]) for r in replicated_rows) @@ -551,6 +691,13 @@ def render(doc: dict[str, Any], document_path: str | None = None) -> str: ) if doc.get("unavailable"): lines.append("Unavailable: " + ", ".join(doc["unavailable"]) + ".") + partial = [ + f"{row.get('lane', '*')}/{row.get('measure', '*')}" + for row in doc.get("run", []) + if row.get("status") == "partial" + ] + if duplication and partial: + lines.append("Partial: " + ", ".join(partial) + ".") if document_path: lines.append(f"Full document: {document_path}") return "\n".join(lines) + "\n" @@ -569,9 +716,12 @@ def main(argv: list[str]) -> int: p_asm.add_argument("--measures", required=True) p_asm.add_argument("--thresholds", required=True) p_asm.add_argument("--excluded") + p_asm.add_argument("--root", default="") p_render = sub.add_parser("render") p_render.add_argument("--document") - sub.add_parser("resummarize") + p_render.add_argument("--rollup-depth", type=int, default=2) + p_res = sub.add_parser("resummarize") + p_res.add_argument("--root", default="") args = parser.parse_args(argv) if args.command == "thresholds": config = _read_json(args.config) @@ -585,15 +735,16 @@ def main(argv: list[str]) -> int: _read_jsonl(args.measures), _read_json(args.thresholds), _read_jsonl(args.excluded), + args.root, ) print(json.dumps(doc, indent=2)) return 0 doc = json.load(sys.stdin) if args.command == "resummarize": - doc["summary"] = summarize(doc.get("measures", [])) + doc["summary"] = summarize(doc.get("measures", []), args.root) print(json.dumps(doc, indent=2)) return 0 - sys.stdout.write(render(doc, getattr(args, "document", None))) + sys.stdout.write(render(doc, getattr(args, "document", None), args.rollup_depth)) return 0 diff --git a/plugins/code-metrics/scripts/test_pathglob.py b/plugins/code-metrics/scripts/test_pathglob.py index 5021c4f047..b36ce24446 100755 --- a/plugins/code-metrics/scripts/test_pathglob.py +++ b/plugins/code-metrics/scripts/test_pathglob.py @@ -29,6 +29,32 @@ def run(*args: str) -> subprocess.CompletedProcess: ) +class RootRelativeTests(unittest.TestCase): + def test_a_cwd_relative_path_is_rebased_onto_the_root(self) -> None: + import os + import tempfile + + with tempfile.TemporaryDirectory() as tmp: + root = os.path.realpath(tmp) + lib = os.path.join(root, "lib") + os.mkdir(lib) + before = os.getcwd() + os.chdir(lib) + try: + self.assertEqual( + pathglob.root_relative("../plugins/a/x.sh", root), "plugins/a/x.sh" + ) + self.assertEqual(pathglob.root_relative("x.sh", root), "lib/x.sh") + self.assertEqual( + pathglob.root_relative(os.path.join(root, "y.sh"), root), "y.sh" + ) + finally: + os.chdir(before) + + def test_without_a_root_the_path_is_only_normalized(self) -> None: + self.assertEqual(pathglob.root_relative("./a\\b.sh", ""), "a/b.sh") + + class TranslateTests(unittest.TestCase): def test_bare_extension_matches_at_any_depth(self) -> None: self.assertTrue(pathglob.matches("*.sh", "a/b/c.sh")) diff --git a/plugins/code-metrics/scripts/test_replica_collapse.py b/plugins/code-metrics/scripts/test_replica_collapse.py index fede6e40a2..fdbd226f26 100755 --- a/plugins/code-metrics/scripts/test_replica_collapse.py +++ b/plugins/code-metrics/scripts/test_replica_collapse.py @@ -71,12 +71,19 @@ def test_copies_under_distinct_carriers_collapse_to_one_labelled_row(self) -> No row("plugins/b/hooks/hook-utils.sh"), row("plugins/a/hooks/hook-utils.sh"), row("plugins/c/hooks/hook-utils.sh"), - row("lib/other.sh", function="main", values={"cyclomatic": 3}, over_reference=[]), + row( + "lib/other.sh", + function="main", + values={"cyclomatic": 3}, + over_reference=[], + ), ] result = run("--registry", str(self.registry), stdin=document(rows)) self.assertEqual(result.returncode, 0, result.stderr) out = json.loads(result.stdout)["measures"] - self.assertEqual([r["file"] for r in out], ["plugins/a/hooks/hook-utils.sh", "lib/other.sh"]) + self.assertEqual( + [r["file"] for r in out], ["plugins/a/hooks/hook-utils.sh", "lib/other.sh"] + ) survivor = out[0] self.assertIn("replicated", survivor["labels"]) self.assertEqual(survivor["replicas"]["count"], 3) @@ -84,7 +91,11 @@ def test_copies_under_distinct_carriers_collapse_to_one_labelled_row(self) -> No self.assertEqual(survivor["replicas"]["path"], "hooks/hook-utils.sh") self.assertEqual( survivor["replicas"]["files"], - ["plugins/a/hooks/hook-utils.sh", "plugins/b/hooks/hook-utils.sh", "plugins/c/hooks/hook-utils.sh"], + [ + "plugins/a/hooks/hook-utils.sh", + "plugins/b/hooks/hook-utils.sh", + "plugins/c/hooks/hook-utils.sh", + ], ) self.assertNotIn("replicas", out[1]) @@ -98,6 +109,30 @@ def test_copies_whose_numbers_differ_stay_separate(self) -> None: self.assertEqual(len(out), 2) self.assertTrue(all("replicas" not in r for r in out)) + def test_a_cluster_line_is_left_to_the_clone_group_reader(self) -> None: + # ` -> ...` names a root canonical for the + # duplication audit; it is never a path-within-plugin, so this pass + # skips it and the plain line beside it still collapses. + registry = Path(self.tmp.name) / "clusters.txt" + registry.write_text( + "lib/hook-utils.sh -> plugins/*/hooks/hook-utils.sh\nhooks/hook-utils.sh\n", + encoding="utf-8", + ) + rows = [ + row("lib/hook-utils.sh"), + row("plugins/a/hooks/hook-utils.sh"), + row("plugins/b/hooks/hook-utils.sh"), + ] + result = run("--registry", str(registry), stdin=document(rows)) + self.assertEqual(result.returncode, 0, result.stderr) + out = json.loads(result.stdout)["measures"] + self.assertEqual( + [r["file"] for r in out], + ["lib/hook-utils.sh", "plugins/a/hooks/hook-utils.sh"], + ) + self.assertEqual(out[1]["replicas"]["count"], 2) + self.assertEqual(out[1]["replicas"]["line"], 2) + def test_a_single_copy_is_not_a_replica(self) -> None: rows = [row("plugins/a/hooks/hook-utils.sh")] result = run("--registry", str(self.registry), stdin=document(rows)) @@ -108,7 +143,11 @@ def test_a_single_copy_is_not_a_replica(self) -> None: def test_prefix_makes_a_subdirectory_run_match_root_relative_lines(self) -> None: rows = [row("a/hooks/hook-utils.sh"), row("b/hooks/hook-utils.sh")] result = run( - "--prefix", "plugins/", "--registry", str(self.registry), stdin=document(rows) + "--prefix", + "plugins/", + "--registry", + str(self.registry), + stdin=document(rows), ) out = json.loads(result.stdout)["measures"] self.assertEqual(len(out), 1) @@ -120,8 +159,16 @@ def test_clone_group_rows_pass_through_untouched(self) -> None: "function": None, "lane": "bash", "instances": [ - {"file": "plugins/a/hooks/hook-utils.sh", "start_line": 1, "end_line": 5}, - {"file": "plugins/b/hooks/hook-utils.sh", "start_line": 1, "end_line": 5}, + { + "file": "plugins/a/hooks/hook-utils.sh", + "start_line": 1, + "end_line": 5, + }, + { + "file": "plugins/b/hooks/hook-utils.sh", + "start_line": 1, + "end_line": 5, + }, ], "values": {"lines": 5, "tokens": 20}, } @@ -130,7 +177,10 @@ def test_clone_group_rows_pass_through_untouched(self) -> None: self.assertEqual(out, [clone, clone]) def test_no_registry_changes_nothing(self) -> None: - rows = [row("plugins/a/hooks/hook-utils.sh"), row("plugins/b/hooks/hook-utils.sh")] + rows = [ + row("plugins/a/hooks/hook-utils.sh"), + row("plugins/b/hooks/hook-utils.sh"), + ] result = run(stdin=document(rows)) self.assertEqual(json.loads(result.stdout)["measures"], rows) diff --git a/plugins/code-metrics/scripts/test_report.py b/plugins/code-metrics/scripts/test_report.py index d52a8c7a7b..8c94189190 100755 --- a/plugins/code-metrics/scripts/test_report.py +++ b/plugins/code-metrics/scripts/test_report.py @@ -982,6 +982,335 @@ def test_resummarize_recomputes_the_summary_after_rows_are_dropped(self) -> None self.assertEqual(out["run"], doc["run"]) +def clone_row(lane: str, first: str, second: str, lines: int, tokens: int = 90) -> dict: + return { + "file": None, + "function": None, + "lane": lane, + "instances": [ + {"file": first, "start_line": 1, "end_line": lines}, + {"file": second, "start_line": 1, "end_line": lines}, + ], + "values": {"lines": lines, "tokens": tokens}, + "over_reference": [], + } + + +def duplication_doc(measures: list[dict], **overrides: object) -> dict: + doc = { + "schema": "code-metrics/v1", + "skill": "audit-duplication", + "status": "complete", + "scope": {"mode": "all", "base": None, "files": 4, "excluded": 0}, + "run": [ + { + "lane": "bash", + "measure": "duplication", + "collector": "jscpd 5.2.0", + "status": "ok", + "reason": None, + "hint": None, + } + ], + "thresholds": [], + "measures": measures, + "summary": {"files": 0, "functions": 0, "over_reference": {}}, + "excluded": [], + "unavailable": [], + } + doc.update(overrides) + return doc + + +def resummarized(doc: dict, *args: str) -> dict: + result = run("resummarize", *args, stdin=json.dumps(doc)) + assert result.returncode == 0, result.stderr + return json.loads(result.stdout) + + +class DuplicationRollupTests(unittest.TestCase): + ROWS = [ + clone_row("bash", "a/x/u.sh", "b/x/u.sh", 20), + clone_row("python", "a/y/v.py", "c/y/v.py", 7, 30), + clone_row("bash", "a/x/w.sh", "d/x/w.sh", 5, 12), + ] + + def test_the_rollups_sum_to_the_totals_and_root_restates_them(self) -> None: + summary = resummarized(duplication_doc(self.ROWS))["summary"] + self.assertEqual(summary["duplicated_lines"], 32) + self.assertEqual( + summary["by_lane"], + { + "bash": {"groups": 2, "duplicated_lines": 25}, + "python": {"groups": 1, "duplicated_lines": 7}, + }, + ) + self.assertEqual( + sum(b["duplicated_lines"] for b in summary["by_lane"].values()), + summary["duplicated_lines"], + ) + self.assertEqual( + summary["by_directory"]["."], {"groups": 3, "duplicated_lines": 32} + ) + + def test_directory_rollups_are_cumulative_over_the_first_instance(self) -> None: + by_directory = resummarized(duplication_doc(self.ROWS))["summary"][ + "by_directory" + ] + # Every group's first instance sits under `a`, so `a` carries all three + # while its children split them; the second instances count nowhere. + self.assertEqual(by_directory["a"], {"groups": 3, "duplicated_lines": 32}) + self.assertEqual(by_directory["a/x"], {"groups": 2, "duplicated_lines": 25}) + self.assertEqual(by_directory["a/y"], {"groups": 1, "duplicated_lines": 7}) + self.assertNotIn("b", by_directory) + self.assertEqual( + set(by_directory), {".", "a", "a/x", "a/y"}, sorted(by_directory) + ) + + def test_root_makes_directory_keys_root_relative(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + rows = [ + clone_row( + "bash", + str(root / "plugins" / "p" / "u.sh"), + str(root / "plugins" / "q" / "u.sh"), + 9, + ) + ] + by_directory = resummarized(duplication_doc(rows), "--root", tmp)[ + "summary" + ]["by_directory"] + self.assertEqual(set(by_directory), {".", "plugins", "plugins/p"}) + + def test_a_document_without_clone_rows_carries_no_rollup_maps(self) -> None: + summary = resummarized(duplication_doc([]))["summary"] + self.assertNotIn("by_lane", summary) + self.assertNotIn("by_directory", summary) + + +class DuplicationRenderTests(unittest.TestCase): + def rendered(self, doc: dict, *args: str) -> str: + result = run("render", *args, stdin=json.dumps(doc)) + self.assertEqual(result.returncode, 0, result.stderr) + return result.stdout + + def test_clone_groups_render_largest_first(self) -> None: + rows = [ + clone_row("bash", "a/small.sh", "b/small.sh", 5, 12), + clone_row("bash", "a/big.sh", "b/big.sh", 20), + clone_row("bash", "a/mid.sh", "b/mid.sh", 5, 40), + ] + out = self.rendered(resummarized(duplication_doc(rows))) + self.assertLess(out.index("a/big.sh"), out.index("a/mid.sh")) + self.assertLess(out.index("a/mid.sh"), out.index("a/small.sh")) + + def test_the_rollup_section_cuts_directories_at_the_depth(self) -> None: + rows = [clone_row("bash", "a/x/z/u.sh", "b/u.sh", 20)] + doc = resummarized(duplication_doc(rows)) + self.assertIn("a/x/z", doc["summary"]["by_directory"]) + out = self.rendered(doc) + self.assertIn("## Rollup", out) + self.assertIn("| bash | 1 | 20 |", out) + self.assertIn("| . | 1 | 20 |", out) + self.assertIn("| a/x | 1 | 20 |", out) + self.assertNotIn("| a/x/z |", out) + self.assertIn("| a/x/z | 1 | 20 |", self.rendered(doc, "--rollup-depth", "3")) + + def test_the_summary_line_counts_files_with_clones(self) -> None: + rows = [clone_row("bash", "a/u.sh", "b/u.sh", 20)] + out = self.rendered(resummarized(duplication_doc(rows))) + self.assertIn("\nFiles with clones: 2.\n", out) + self.assertNotIn("Functions:", out) + + def test_an_empty_excluded_list_is_stated_with_its_reason(self) -> None: + rows = [clone_row("bash", "a/u.sh", "b/u.sh", 20)] + out = self.rendered(resummarized(duplication_doc(rows))) + self.assertIn( + "Excluded by a sanctioned-replication registry: 0 (no registry configured, or " + "none matched).", + out, + ) + doc = resummarized(duplication_doc(rows)) + doc["excluded"] = [{"registry": "r.txt", "line": 3, "path": "u.sh"}] + self.assertIn( + "Excluded by a sanctioned-replication registry: 1.", self.rendered(doc) + ) + + def test_no_detector_prints_one_headline_with_the_hint(self) -> None: + hint = "jscpd: https://github.com/kucherenko/jscpd (npm install -g jscpd)" + doc = duplication_doc( + [], + status="empty", + run=[ + { + "lane": lane, + "measure": "duplication", + "collector": None, + "status": "unavailable", + "reason": "jscpd: not on PATH", + "hint": hint, + } + for lane in ("bash", "python") + ], + unavailable=["bash/duplication", "python/duplication"], + ) + out = self.rendered(doc) + self.assertEqual(out.count("No clone detector ran in any lane"), 1) + self.assertEqual(out.count("npm install -g jscpd"), 1) + self.assertIn("/code-metrics:setup", out) + self.assertEqual(out.count("| unavailable | jscpd: not on PATH |"), 2) + self.assertLess(out.index("No clone detector"), out.index("## Coverage")) + + def test_a_not_applicable_lane_does_not_hide_the_no_detector_headline( + self, + ) -> None: + # The `other` lane carries a `not-applicable` duplication row on every + # run that has a file outside the language lanes; it is not a probe + # that failed, so it must not defeat the all-unavailable check. + hint = "jscpd: https://github.com/kucherenko/jscpd (npm install -g jscpd)" + doc = duplication_doc( + [], + status="empty", + run=[ + { + "lane": "bash", + "measure": "duplication", + "collector": None, + "status": "unavailable", + "reason": "jscpd: not on PATH", + "hint": hint, + }, + { + "lane": "other", + "measure": "duplication", + "collector": None, + "status": "not-applicable", + "reason": "no collector covers this lane", + "hint": None, + }, + ], + unavailable=["bash/duplication"], + ) + out = self.rendered(doc) + self.assertEqual(out.count("No clone detector ran in any lane"), 1) + self.assertEqual(out.count("npm install -g jscpd"), 1) + + def test_only_not_applicable_rows_print_no_headline(self) -> None: + doc = duplication_doc( + [], + status="empty", + run=[ + { + "lane": "other", + "measure": "duplication", + "collector": None, + "status": "not-applicable", + "reason": "no collector covers this lane", + "hint": None, + } + ], + ) + self.assertNotIn("No clone detector ran", self.rendered(doc)) + + def test_a_lane_that_skipped_every_file_is_partial_not_empty(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + d = Path(tmp) + result = run( + "assemble", + "--skill", + "audit-duplication", + "--scope", + write(d, "s.json", json.dumps({"mode": "all", "files": 1})), + "--run", + write( + d, + "r.jsonl", + jsonl( + [ + { + "lane": "bash", + "measure": "duplication", + "collector": "jscpd 5.2.0", + "status": "partial", + "reason": "1 of 1 files skipped by duplication.max_size 1mb / max_lines none", + "hint": None, + } + ] + ), + ), + "--measures", + write(d, "m.jsonl", ""), + "--thresholds", + write(d, "t.json", "[]"), + ) + self.assertEqual(result.returncode, 0, result.stderr) + doc = json.loads(result.stdout) + self.assertEqual(doc["status"], "partial") + out = self.rendered(doc) + self.assertNotIn("Measured nothing", out) + self.assertIn("Status: partial", out) + self.assertIn("\nPartial: bash/duplication.\n", out) + + def test_a_size_document_renders_as_before(self) -> None: + doc = { + "schema": "code-metrics/v1", + "skill": "audit-size", + "status": "partial", + "scope": {"mode": "all", "base": None, "files": 1, "excluded": 0}, + "run": [ + { + "lane": "python", + "measure": "size", + "collector": "scc 3.4.0", + "status": "partial", + "reason": "1 of 2 files unreadable", + "hint": None, + } + ], + "thresholds": [], + "measures": [ + { + "file": "a.py", + "function": None, + "lane": "python", + "values": {"lines_non_blank": 12}, + "over_reference": ["file_lines"], + } + ], + "summary": { + "files": 1, + "functions": 0, + "over_reference": {"file_lines": 1}, + }, + "excluded": [], + "unavailable": [], + } + out = self.rendered(doc) + self.assertEqual( + out, + "# code-metrics: audit-size\n" + "\n" + "Status: partial. Scope: all, 1 file(s).\n" + "\n" + "## Coverage of this run\n" + "\n" + "| Lane | Measure | Collector | Status | Reason |\n" + "|---|---|---|---|---|\n" + "| python | size | scc 3.4.0 | partial | 1 of 2 files unreadable |\n" + "\n" + "## Measures\n" + "\n" + "| File | Function | Lane | Labels | lines_non_blank | Over reference |\n" + "|---|---|---|---|---|---|\n" + "| a.py | | python | | 12 | file_lines |\n" + "\n" + "## Summary\n" + "\n" + "Files: 1. Over reference: file_lines 1.\n", + ) + + def render_doc(measures: list[dict], thresholds_: list[dict] | None = None, **scope): doc = { "schema": "code-metrics/v1", diff --git a/plugins/code-metrics/skills/audit-duplication/SKILL.md b/plugins/code-metrics/skills/audit-duplication/SKILL.md index 3e4ef00e47..3297681a86 100644 --- a/plugins/code-metrics/skills/audit-duplication/SKILL.md +++ b/plugins/code-metrics/skills/audit-duplication/SKILL.md @@ -1,5 +1,5 @@ --- -description: "Measure duplicated code as clone groups over the changed files, a path, or the whole tree: each group's duplicated lines and tokens with every instance's file and line range, per lane (TypeScript/JavaScript, Python, Bash, Go, C#) from whichever clone detector already resolves. Replication the target repository declares about itself, a file vendored into several plugins and listed by path-within-plugin in a sanctioned-replication registry, is subtracted from the total and reported as an exclusion naming the registry line rather than as debt, and the report emits no finding, no severity, and no exit-code gate. Use when: 'is this duplicated', 'find copy-paste code', 'clone detection', 'duplication report', 'how much of this change is copied', 'DRY check', 'redundant code', 'duplicated lines in the diff'; for lines per file use /code-metrics:audit-size, and for what a duplication number can and cannot support use /code-metrics:principles." +description: "Measure duplicated code as clone classes over the changed files, a path, or the whole tree: each class's duplicated lines and tokens with every instance's file and line range, rolled up per lane (TypeScript/JavaScript, Python, Bash, Go, C#) and per directory, from whichever clone detector already resolves. Replication the target repository declares about itself in a sanctioned-replication registry (a path-within-plugin or a canonical-to-copies cluster line) is subtracted from the total and reported as an exclusion naming the registry line rather than as debt; the report emits no finding, no severity, and no exit-code gate. Use when: 'is this duplicated', 'find copy-paste code', 'clone detection', 'duplication report', 'how much of this change is copied', 'DRY check', 'redundant code', 'duplicated lines in the diff'; for lines per file use /code-metrics:audit-size, and for what a duplication number can and cannot support use /code-metrics:principles." argument-hint: "[--json] [--all] [--base ] [--registry ] [...]" user-invocable: true disable-model-invocation: false @@ -44,9 +44,13 @@ continues. This plugin never installs, downloads, or `npx`-fetches a detector. ``` Present the markdown report as printed. It opens with the scope and a "Coverage of this run" -table (lane, collector, status, reason), then one row per clone group listing every instance as -`file:start-end`, then the summary line with the duplicated-line total and how many groups a -registry excluded. Keep the `--json` document when the numbers feed a comparison: +table (lane, collector, status, reason), then one row per clone group, largest first, listing +every instance as `file:start-end`, then a rollup per lane and per directory, then the summary +lines: files with clones, the duplicated-line total, how many groups a registry excluded, and +which lanes were partial. When the report opens with `No clone detector ran in any lane`, offer +the user the install command that headline carries (`npm install -g jscpd`, or a devDependency) +and run it only when they confirm; never install silently and never `npx`-fetch it. Keep the +`--json` document when the numbers feed a comparison: `/verification:measure metrics` consumes it when the `verification` plugin is installed (treat a report whose `status` is `empty` on either side as INCONCLUSIVE); otherwise keep the JSON beside your notes and compare by hand. @@ -55,9 +59,24 @@ your notes and compare by hand. - A clone group is reported beside no reference. There is no configured bar for duplication in this plugin and no standard sets one, so nothing is ever counted as `over_reference`. -- `summary.duplicated_lines` counts each group once, using the length of the group, not the sum - over its instances: two copies of a 41-line block are 41 duplicated lines, not 82. The count is - what survived the registries. +- A group is a clone class, not a detector pair. `jscpd` and PMD CPD report clones as pairs, so + seventeen identical copies arrive as sixteen two-instance rows; this skill merges rows that + share an instance with an identical file and line range into one row per class before it + counts anything. Copies that share only part of a fragment are named with different ranges and + stay separate groups: the merge joins on identity, never on overlap, so a class is never wider + than what the detector called identical. +- `summary.duplicated_lines` counts each class once, using the length of the class, not the sum + over its instances: three copies of a 41-line block are 41 duplicated lines, not 82 or 123. The + count is what survived the registries. +- `summary.by_lane` and `summary.by_directory` roll the surviving classes up: each maps to + `{groups, duplicated_lines}`, the directory map for `.` and every ancestor of each class's first + instance. A class counts once under every ancestor, so a parent includes its children and the + directory rows cannot be summed; `by_directory["."]` and the per-lane sum both restate the + totals. The markdown lists directories to `duplication.rollup_depth`; the JSON carries all. +- A file larger than `duplication.max_size` (or longer than `duplication.max_lines`, when set) is + left out of the scan and never silently dropped: the lane's run row is `partial` with how many + files were skipped and the largest one, the document is `partial`, and the summary carries a + `Partial:` line naming the lane. - The registry is an **exclusion**, not a suppression: it is derived from the target repository's own declaration that those copies are deliberate, so no suppression record is involved and the excluded groups stay in the document under `excluded[]` with the registry path, the 1-based line @@ -66,9 +85,10 @@ your notes and compare by hand. directory are ordinary duplication and stay. - A value the detector did not produce is `null`, never `0`: `dupl` reports no token count, so its rows carry `tokens: null`. -- `status` is `complete` when every lane in scope was measured, `partial` when one was not, and - `empty` when nothing was; a run that measured nothing prints "Measured nothing" and states no - duplication figure at all, which is not the same as zero duplication. +- `status` is `complete` when every lane in scope was measured, `partial` when one was not or + when a cap left files out of one, and `empty` when nothing was; a run that measured nothing + prints "Measured nothing" and states no duplication figure at all, which is not the same as zero + duplication. A lane that skipped every file is still `partial`: its row says what was skipped. - Exit 0 whenever a report was produced, including an `empty` one; exit 2 for a usage error such as a named registry or scope path that does not exist; exit 3 when a detector resolved but produced nothing parseable, with its stderr in the run table. A detector's own non-zero exit is @@ -79,17 +99,34 @@ your notes and compare by hand. Everything tunable resolves through `.claude/code-metrics.yaml` (user-global, team, local overlay; per-key override; keys in `${CLAUDE_PLUGIN_ROOT}/reference/config.md`): `duplication.min_tokens` (default 50), `duplication.min_lines` (default 5), -`duplication.ignore` (globs handed to the detector's own ignore option), and -`scope.registries` (sanctioned-replication registries, each path relative to the repository root, -each also nameable on the command line with `--registry`, and read by every audit in this plugin; -`duplication.registries` is the older name and still resolves when the scope-level list is -empty). `/code-metrics:setup` writes the team file and probes the collectors. - -This script exports the three tunables to the collector adapters as -`CODE_METRICS_DUP_MIN_TOKENS`, `CODE_METRICS_DUP_MIN_LINES`, and `CODE_METRICS_DUP_IGNORE`, which -is the only channel an adapter reads them through. `jscpd` passes all three to the tool; -`dupl` and `cpd` have no minimum-lines or ignore-glob option, so their adapters apply the -minimum after parsing and report the ignore globs as unused. +`duplication.ignore` (globs handed to the detector's own ignore option), `duplication.max_size` +(default `1mb`, binary units; a larger file is left out of the scan and reported), `duplication.max_lines` +(default `null`, no line cap), `duplication.rollup_depth` (default 2, how deep the markdown +per-directory rollup lists), and `scope.registries` (sanctioned-replication registries, each path +relative to the repository root, each also nameable on the command line with `--registry`, and +read by every audit in this plugin; `duplication.registries` is the older name and still resolves +when the scope-level list is empty). A cap of `null` or `0` means no cap. `/code-metrics:setup` +writes the team file and probes the collectors. + +A registry line has one of two shapes. A plain line is one path-within-plugin, taken whole with +any spaces: a class is excluded when every instance ends with that path and the copies sit in +distinct carrying directories. A cluster line, ` -> ...`, names a root-relative +canonical copy and the plugin paths or gitignore-style globs that carry it +(`lib/hook-utils.sh -> plugins/*/hooks/hook-utils.sh`): a class is excluded when every instance is +the canonical or matches a member and the instances' directories are pairwise distinct. Instance +paths are compared root-relative, so a run from a subdirectory matches the same lines, and the +first matching line in file order wins. + +This script exports the five tunables to the collector adapters as +`CODE_METRICS_DUP_MIN_TOKENS`, `CODE_METRICS_DUP_MIN_LINES`, `CODE_METRICS_DUP_IGNORE`, +`CODE_METRICS_DUP_MAX_LINES`, and `CODE_METRICS_DUP_MAX_SIZE`, which is the only channel an +adapter reads them through. `jscpd` passes the first three to the tool and applies the two caps +itself before the tool runs; the `dupl` and `cpd` adapters apply the minimum after parsing, report +the ignore globs as unused, and scan every file in scope. Which options each tool honours, what its +own defaults are, and why the caps are applied here rather than passed through are tool facts that +move with the tools, so they are not restated here: the duplication rows of +`${CLAUDE_PLUGIN_ROOT}/reference/collectors.md` carry each claim with its basis, the date it was +verified, and the upstream event that obliges a recheck. `cpd` (PMD) sits after `jscpd` on `${CLAUDE_PLUGIN_ROOT}/scripts/collector-ladder.tsv` for every lane but Bash, so it runs only when `jscpd` does not resolve and `pmd` does. A repository that @@ -124,7 +161,11 @@ overrides are validated against the ladder file and an unknown name is dropped w - Clone detection compares the files in scope with each other. A default-scope run sees only the changed files, so a block copied from a file the change did not touch is not found; use `--all` or name both paths when that is the question. -- `jscpd` reports pairs, so seventeen identical copies arrive as sixteen two-instance groups - rather than one seventeen-instance group; the registry excludes each of them on the same line. +- The pair merge is exact only for byte-identical copies. A class whose copies drifted by a line + is reported as the detector saw it: the identical span as one class, and the drifted copy's + shorter overlap as a second group naming the same file with a different range. +- Two versions of one detector can tokenize the same tree differently and so report different + class counts (the jscpd rows of `${CLAUDE_PLUGIN_ROOT}/reference/collectors.md` record the + verified case); compare runs made with one detector version, never across a version boundary. - Lowering `duplication.min_tokens` finds more and smaller clones, most of them boilerplate the language forces; the defaults are the detector's own conservative pair. diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh b/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh index b2ab3d8ef7..ae052f251d 100755 --- a/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh +++ b/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh @@ -7,10 +7,12 @@ # # Prints the markdown report; `--json` prints the `code-metrics/v1` document # instead. Scope, lanes, and the collector ladder are the dispatcher's -# (scripts/dispatch.sh in the plugin root); this script owns `--registry` and -# the duplication tunables it exports for the collector adapters +# (scripts/dispatch.sh in the plugin root); this script owns the merge of the +# detector's clone pairs into clone classes (cluster-clones.py), `--registry`, +# and the duplication tunables it exports for the collector adapters # (CODE_METRICS_DUP_MIN_TOKENS, CODE_METRICS_DUP_MIN_LINES, -# CODE_METRICS_DUP_IGNORE, from `duplication.*` in the resolved config). +# CODE_METRICS_DUP_IGNORE, CODE_METRICS_DUP_MAX_LINES, CODE_METRICS_DUP_MAX_SIZE, +# from `duplication.*` in the resolved config; a null or 0 cap exports empty). # Registries come from every `--registry` plus `duplication.registries`, each # resolved against the repository root; a named registry that does not exist is # a usage error. Exit codes are the dispatcher's: 0 report produced, 2 usage @@ -21,6 +23,7 @@ SCRIPT_DIR="$(cd "${BASH_SOURCE[0]%/*}" && pwd)" PLUGIN_ROOT="${CLAUDE_PLUGIN_ROOT:-$(cd "$SCRIPT_DIR/../../.." && pwd)}" DISPATCH="$PLUGIN_ROOT/scripts/dispatch.sh" REPORT="$PLUGIN_ROOT/scripts/report.py" +CLUSTER="$SCRIPT_DIR/cluster-clones.py" FILTER="$SCRIPT_DIR/registry-filter.py" JSON=0 @@ -50,7 +53,7 @@ while [[ $# -gt 0 ]]; do shift 2 ;; --help | -h) - sed -n '2,17p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' >&2 + sed -n '2,19p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' >&2 exit 0 ;; *) @@ -81,7 +84,8 @@ if [[ -z "$CONFIG" ]]; then --home "${CODE_METRICS_HOME:-${HOME:-/}}" >"$CONFIG" || exit 2 fi -# The three tunables, then the registries from the resolver's own format +# Six tunables (a cap of null or 0 is exported empty, which the adapter reads +# as "no cap"), then the registries from the resolver's own format # (`scope.registries`, or `duplication.registries` as its older name), so this # script and the dispatcher read the same list the same way. mapfile -t DUP < <("${PY[@]}" -c ' @@ -95,18 +99,34 @@ def number(key, fallback): return value if isinstance(value, int) and not isinstance(value, bool) else fallback +def cap(key): + value = section.get(key) + if value is None or isinstance(value, bool): + return "" + if isinstance(value, (int, float)): + return str(int(value)) if value > 0 else "" + text = str(value).strip() + return "" if text in ("", "0") else text + + print(number("min_tokens", 50)) print(number("min_lines", 5)) ignore = section.get("ignore") print(",".join(str(item) for item in ignore) if isinstance(ignore, list) else "") +print(cap("max_lines")) +print(cap("max_size")) +print(number("rollup_depth", 2)) ' "$CONFIG") -if [[ ${#DUP[@]} -lt 3 ]]; then +if [[ ${#DUP[@]} -lt 6 ]]; then echo "audit-duplication.sh: the resolved configuration could not be read" >&2 exit 2 fi export CODE_METRICS_DUP_MIN_TOKENS="${DUP[0]}" export CODE_METRICS_DUP_MIN_LINES="${DUP[1]}" export CODE_METRICS_DUP_IGNORE="${DUP[2]}" +export CODE_METRICS_DUP_MAX_LINES="${DUP[3]}" +export CODE_METRICS_DUP_MAX_SIZE="${DUP[4]}" +ROLLUP_DEPTH="${DUP[5]}" if ! "${PY[@]}" "$PLUGIN_ROOT/scripts/resolve-config.py" --from-json "$CONFIG" --format registries >"$WORK/registries"; then echo "audit-duplication.sh: the configured registries could not be read (see the message above)" >&2 exit 2 @@ -137,10 +157,12 @@ bash "$DISPATCH" audit-duplication --measures duplication --config "$CONFIG" ${P rc=$? [[ $rc -eq 0 || $rc -eq 3 ]] || exit "$rc" -# Exclude the declared replication, recompute the totals from what survived, -# then state the zero the recomputation drops when every group was excluded. -"${PY[@]}" "$FILTER" "${FILTER_ARGS[@]}" <"$WORK/report.json" >"$WORK/filtered.json" || exit 2 -"${PY[@]}" "$REPORT" resummarize <"$WORK/filtered.json" >"$WORK/summed.json" || exit 2 +# Merge the pairs the detector reports into clone classes, exclude the declared +# replication, recompute the totals from what survived, then state the zero the +# recomputation drops when every group was excluded. +"${PY[@]}" "$CLUSTER" --root "$ROOT" <"$WORK/report.json" >"$WORK/clustered.json" || exit 2 +"${PY[@]}" "$FILTER" "${FILTER_ARGS[@]}" <"$WORK/clustered.json" >"$WORK/filtered.json" || exit 2 +"${PY[@]}" "$REPORT" resummarize --root "$ROOT" <"$WORK/filtered.json" >"$WORK/summed.json" || exit 2 "${PY[@]}" "$FILTER" --zero-floor --root "$ROOT" <"$WORK/summed.json" >"$WORK/final.json" || exit 2 if [[ $JSON -eq 1 ]]; then @@ -148,9 +170,9 @@ if [[ $JSON -eq 1 ]]; then else # shellcheck source=../../../scripts/persist-report.sh source "$PLUGIN_ROOT/scripts/persist-report.sh" - render_args=() + render_args=(--rollup-depth "$ROLLUP_DEPTH") if document="$(cm_persist_report audit-duplication "$WORK/final.json")"; then - render_args=(--document "$document") + render_args+=(--document "$document") fi "${PY[@]}" "$REPORT" render "${render_args[@]}" <"$WORK/final.json" || exit 2 fi diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh b/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh index 9a2cdcf681..79ebd93a0f 100755 --- a/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh +++ b/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh @@ -9,7 +9,11 @@ # adapter passes and exits 1, the reporting exit code the contract says is not # a failure (design T13; no executable is committed). The fixture cluster is # scripts/fixtures/sources/cluster/{alpha,beta}/shared/shared-utils.sh and the -# registry that sanctions it is scripts/fixtures/registry/cluster.txt. +# registry that sanctions it is scripts/fixtures/registry/cluster.txt. The +# three-copy cases swap in the captures jscpd-aligned3.json and +# jscpd-offset3.json, real jscpd 5.2.0 runs over +# scripts/fixtures/clone-classes/{aligned,offset} rewritten to repo-relative +# names. # # The last case is the Brief's own: this repository's real # plugins/*/hooks/hook-utils.sh cluster against @@ -63,7 +67,7 @@ export CODE_METRICS_REPORT_DIR="$STUBS/reports" cat >"$STUBS/jscpd" <<'STUB' #!/usr/bin/env bash if [[ "${1:-}" == "--version" ]]; then - printf 'jscpd 5.1.2\n' + printf 'jscpd 5.2.0\n' exit 0 fi [[ -z "${CM_TEST_ARGV_LOG:-}" ]] || printf '%s\n' "$*" >>"$CM_TEST_ARGV_LOG" @@ -147,19 +151,45 @@ assert_eq "a missing --registry exits 2" 2 "$?" # 6. The configured tunables reach the collector's command line. "$PY" "$PLUGIN_ROOT/scripts/resolve-config.py" --ladder "$PLUGIN_ROOT/scripts/collector-ladder.tsv" --home "$WORK" >"$WORK/base.json" 2>/dev/null -"$PY" -c 'import json,sys; d=json.load(open(sys.argv[1])); d["duplication"]["min_tokens"] = 77; d["duplication"]["min_lines"] = 9; d["duplication"]["ignore"] = ["**/vendor/**"]; print(json.dumps(d))' "$WORK/base.json" >"$WORK/tuned.json" +"$PY" -c 'import json,sys; d=json.load(open(sys.argv[1])); d["duplication"]["min_tokens"] = 77; d["duplication"]["min_lines"] = 9; d["duplication"]["ignore"] = ["**/vendor/**"]; d["duplication"]["max_size"] = "8kb"; d["duplication"]["max_lines"] = 0; print(json.dumps(d))' "$WORK/base.json" >"$WORK/tuned.json" CM_TEST_ARGV_LOG="$WORK/argv.log" PATH="$STUBS:$EMPTY_PATH" bash "$SCRIPT" --json --all "$CLUSTER" --config "$WORK/tuned.json" >/dev/null 2>&1 assert_eq "the tuned run exits 0" 0 "$?" argv="$(cat "$WORK/argv.log" 2>/dev/null)" assert_contains "min_tokens reaches the collector" "$argv" "--min-tokens 77" assert_contains "min_lines reaches the collector" "$argv" "--min-lines 9" assert_contains "the ignore globs reach the collector" "$argv" "--ignore **/vendor/**" +assert_contains "max_size reaches the collector one byte above the bound" "$argv" "--max-size 8193" +assert_contains "a max_lines of 0 means no cap and reaches the collector as the explicit large value" "$argv" "--max-lines 2147483647" -# 7. --help prints the usage without running anything. +# 7. Three byte-identical copies are one clone class, its lines counted once. +# jscpd pairs each later copy with the first, so the capture holds two pairs +# that name the same instance of copy `a`. +ALIGNED="$FIXTURES/clone-classes/aligned" +out="$(CM_TEST_CAPTURE="$REPO_ROOT/$FIXTURES/tool-output/jscpd-aligned3.json" PATH="$STUBS:$EMPTY_PATH" bash "$SCRIPT" --json --all "$ALIGNED")" +assert_eq "the aligned three-copy run exits 0" 0 "$?" +if printf '%s' "$out" | "$PY" -c 'import json,sys; d=json.load(sys.stdin); assert d["summary"]["clone_groups"] == 1 and d["summary"]["duplicated_lines"] == 41, d["summary"]; row = d["measures"][0]; assert len(row["instances"]) == 3 and "clustered" in row["labels"], row' 2>/dev/null; then + pass "three aligned copies are one clone class with the lines counted once" +else + fail "three aligned copies are one clone class with the lines counted once" "clone_groups 1, duplicated_lines 41, three instances" "$(printf '%s' "$out" | head -c 600)" +fi + +# 8. A third copy that shares only part of the fragment stays its own group: +# the two pairs name copy `c1` with different ranges, and overlap is not +# identity. +OFFSET="$FIXTURES/clone-classes/offset" +out="$(CM_TEST_CAPTURE="$REPO_ROOT/$FIXTURES/tool-output/jscpd-offset3.json" PATH="$STUBS:$EMPTY_PATH" bash "$SCRIPT" --json --all "$OFFSET")" +assert_eq "the offset three-copy run exits 0" 0 "$?" +if printf '%s' "$out" | "$PY" -c 'import json,sys; d=json.load(sys.stdin); assert d["summary"]["clone_groups"] == 2 and d["summary"]["duplicated_lines"] == 58, d["summary"]; assert all(len(r["instances"]) == 2 and "clustered" not in r["labels"] for r in d["measures"]), d["measures"]' 2>/dev/null; then + pass "a partial third copy stays a second clone group" +else + fail "a partial third copy stays a second clone group" "clone_groups 2, duplicated_lines 58, two instances each" "$(printf '%s' "$out" | head -c 600)" +fi + +# 9. --help prints the usage without running anything. bash "$SCRIPT" --help 2>&1 | grep -q 'audit-duplication.sh \[--json\]' assert_eq "--help prints usage" 0 "$?" -# 8. The Brief's case: this repository's own vendored hook-utils cluster. +# 10. The Brief's case: this repository's own vendored hook-utils cluster. # The jscpd on PATH has to be a working detector, not another suite's stub or # a replaying fake: the probe copies one fixture into two directories under a # name nothing else uses and requires the report to name it back. diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/cluster-clones.py b/plugins/code-metrics/skills/audit-duplication/scripts/cluster-clones.py new file mode 100755 index 0000000000..007b5fb774 --- /dev/null +++ b/plugins/code-metrics/skills/audit-duplication/scripts/cluster-clones.py @@ -0,0 +1,165 @@ +#!/usr/bin/env python3 +"""Merge the clone pairs a detector reports into clone classes. + + cluster-clones.py [--root ] [< report.json] + +Reads a `code-metrics/v1` document on stdin and prints it back with the +two-instance clone-group rows that share an identical instance merged into one +row per clone class. jscpd (both majors) and PMD CPD report a clone as a PAIR, +so N byte-identical copies of one fragment arrive as N-1 rows that all name +the same instance of the first copy, and a summary derived from those rows +would count the fragment's lines N-1 times. A clone class is the union of +every pair that shares a code portion (Roy and Cordy 2007 s.6, citing Rieger, +Ducasse and Lanza 2004; Roy, Cordy and Koschke 2009 aggregate pairs into +classes in post-processing), and that closure is exact for the byte-identical +clones a token detector reports. + +Two rows join when they share an instance with identical `(file, start_line, +end_line)` and equal `values.lines`. Overlap is not enough: a pair whose +shared file is named with a different range (a third copy that carries only +part of the fragment) stays its own group, so a class is never widened past +what the detector said was identical. The merged row keeps the first row's +`values`, so the fragment's lines count once, carries the union of the +instances sorted by `(file, start_line)`, and appends `clustered` to `labels`. +The sort compares each file made relative to `--root` (the instance keeps the +path the detector gave it), so the first instance, and the directory the +report's rollup attributes the class to, is the same whichever directory the +run started from; without `--root` the paths sort as given. +A row with three or more instances is already a class and passes through, as +does every row without `instances`, and every row keeps its position. Rows +join whatever their `collector`, so a pair another detector reported merges +too. `summary` is left alone: the caller recomputes it with `report.py +resummarize`. + +Exit 0 when the document was printed, 2 when stdin is not a JSON document. +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "scripts")) +from pathglob import root_relative # noqa: E402 + +MIN_PYTHON = (3, 9) +LABEL = "clustered" + + +def instance_key(instance: dict[str, Any]) -> tuple[str, Any, Any]: + return ( + str(instance.get("file", "")).replace("\\", "/"), + instance.get("start_line"), + instance.get("end_line"), + ) + + +def _sort_key(instance: dict[str, Any], root: str) -> tuple[str, int]: + start = instance.get("start_line") + return ( + root_relative(instance_key(instance)[0], root), + start if isinstance(start, int) else -1, + ) + + +def _is_pair(row: dict[str, Any]) -> bool: + instances = row.get("instances") + return isinstance(instances, list) and len(instances) == 2 + + +def cluster(measures: list[dict[str, Any]], root: str = "") -> list[dict[str, Any]]: + """Return `measures` with pair rows that share an identical instance merged.""" + parent: dict[int, int] = { + index: index for index, row in enumerate(measures) if _is_pair(row) + } + + def find(index: int) -> int: + while parent[index] != index: + parent[index] = parent[parent[index]] + index = parent[index] + return index + + def union(left: int, right: int) -> None: + left, right = find(left), find(right) + if left != right: + # The lower index stays the root, so a class is emitted where its + # first pair stood and its `values` are that first pair's. + parent[max(left, right)] = min(left, right) + + seen: dict[tuple[Any, ...], int] = {} + for index in parent: + row = measures[index] + lines = (row.get("values") or {}).get("lines") + for instance in row["instances"]: + key = (instance_key(instance), lines) + if key in seen: + union(seen[key], index) + else: + seen[key] = index + + members: dict[int, list[int]] = {} + for index in parent: + members.setdefault(find(index), []).append(index) + + output: list[dict[str, Any]] = [] + for index, row in enumerate(measures): + if index not in parent: + output.append(row) + continue + leader = find(index) + if leader != index: + continue + group = members[leader] + if len(group) == 1: + output.append(row) + continue + instances: dict[tuple[str, Any, Any], dict[str, Any]] = {} + for member in group: + for instance in measures[member]["instances"]: + instances.setdefault(instance_key(instance), instance) + merged = dict(row) + merged["instances"] = sorted( + instances.values(), key=lambda instance: _sort_key(instance, root) + ) + labels = [str(label) for label in row.get("labels") or []] + if LABEL not in labels: + labels.append(LABEL) + merged["labels"] = labels + output.append(merged) + return output + + +def main(argv: list[str]) -> int: + root = "" + if argv == ["--root"] or (argv and argv[0] != "--root") or len(argv) > 2: + print("usage: cluster-clones.py [--root ] < report.json", file=sys.stderr) + return 2 + if argv: + root = argv[1] + try: + document = json.load(sys.stdin) + except (json.JSONDecodeError, ValueError) as exc: + print( + f"cluster-clones.py: stdin is not a JSON document ({exc})", file=sys.stderr + ) + return 2 + if not isinstance(document, dict): + print("cluster-clones.py: stdin is not a JSON object", file=sys.stderr) + return 2 + measures = document.get("measures") + if isinstance(measures, list): + document["measures"] = cluster(measures, root) + print(json.dumps(document, indent=2)) + return 0 + + +if __name__ == "__main__": + if sys.version_info < MIN_PYTHON: + print( + "cluster-clones.py needs Python %d.%d or later" % MIN_PYTHON, + file=sys.stderr, + ) + sys.exit(2) + sys.exit(main(sys.argv[1:])) diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py b/plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py index fe5993b4c7..d3147e09ef 100755 --- a/plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py +++ b/plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py @@ -10,20 +10,33 @@ of a finding, so no suppression record is involved and the excluded groups stay visible in the document. -A registry is a text file with one path-within-plugin per line, `#` comments -and blank lines ignored (this repository's own -`scripts/cross-plugin-source-registry.txt` is the shape). A clone group is -dropped when one registry line accounts for EVERY instance: each instance's -path, relative to `--root` and written with forward slashes, is that line or -ends with `/` plus that line, and the prefixes in front of that suffix are all -distinct, so the copies sit in different carrying directories. Two clones -inside one directory are ordinary duplication and stay. +A registry is a text file, `#` comments and blank lines ignored (this +repository's own `scripts/cross-plugin-source-registry.txt` is the shape), with +two kinds of line: + +- A plain line is one path-within-plugin, taken whole, spaces included. It + sanctions a group when EVERY instance's path is that line or ends with `/` + plus that line, and the prefixes in front of that suffix are all distinct, + so the copies sit in different carrying directories. +- A line containing ` -> ` is a cluster line: the text before the arrow is the + root-relative canonical path, the whitespace-separated tokens after it are + members, each a literal root-relative path or a gitignore-style glob + (`plugins/*/hooks/hook-utils.sh`, matched by the plugin's `pathglob`). It + sanctions a group when every instance is the canonical path or matches one + member, and the instances' directories are pairwise distinct. + +Two clones inside one directory are ordinary duplication under either rule and +stay. Every instance path is compared root-relative: a cwd-relative path is +joined onto the working directory and taken relative to `--root`, so a run from +a subdirectory (where the dispatcher names files `../../lib/x.sh`) matches the +same lines a run from the root does. Lines are tried in file order and the +first matching line wins. Each dropped group is appended to `excluded[]` as `{"registry", "line", "path", "instances"}`, naming the registry file, the 1-based line number, and -the line's text that sanctioned it. Rows without `instances` pass through -untouched, and `summary` is left alone: the caller recomputes it with -`report.py resummarize`. +the line's text that sanctioned it (for a cluster line, the whole line). Rows +without `instances` pass through untouched, and `summary` is left alone: the +caller recomputes it with `report.py resummarize`. `--zero-floor` is the pass the caller runs AFTER that recomputation, with no registries: it states `duplicated_lines: 0` and `clone_groups: 0` when a @@ -41,41 +54,50 @@ import json import os import sys +from pathlib import Path from typing import Any +sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "scripts")) +from pathglob import matches as glob_matches # noqa: E402 +from pathglob import root_relative # noqa: E402 + MIN_PYTHON = (3, 9) +CLUSTER_MARKER = " -> " + +# (line number, line text, canonical path or plain token, members). A plain +# line has no members. +Entry = tuple[int, str, str, list[str]] + +def _clean(path: str) -> str: + return path.replace("\\", "/").lstrip("/") -def read_registry(path: str) -> list[tuple[int, str]]: - entries: list[tuple[int, str]] = [] + +def read_registry(path: str) -> list[Entry]: + entries: list[Entry] = [] with open(path, encoding="utf-8") as handle: for number, raw in enumerate(handle, 1): line = raw.strip() if not line or line.startswith("#"): continue - entries.append((number, line.replace("\\", "/").lstrip("/"))) + if CLUSTER_MARKER in line: + canonical, _, rest = line.partition(CLUSTER_MARKER) + members = [_clean(token) for token in rest.split()] + entries.append((number, line, _clean(canonical.strip()), members)) + else: + entries.append((number, line, _clean(line), [])) return entries def relative(path: str, root: str) -> str: - path = (path or "").replace("\\", "/") - if os.path.isabs(path) and root: - try: - path = os.path.relpath(path, root).replace("\\", "/") - except ValueError: - return path - while path.startswith("./"): - path = path[2:] - return path - - -def sanctions(entry: str, instances: list[dict[str, Any]], root: str) -> bool: - """True when this registry line accounts for every instance of the group.""" - if len(instances) < 2: - return False + """The instance path root-relative, with forward slashes.""" + return root_relative(path, root) + + +def sanctions_plain(entry: str, paths: list[str]) -> bool: + """True when this plain line accounts for every instance of the group.""" prefixes = set() - for instance in instances: - path = relative(str(instance.get("file", "")), root) + for path in paths: if path == entry: prefix = "" elif path.endswith("/" + entry): @@ -88,9 +110,34 @@ def sanctions(entry: str, instances: list[dict[str, Any]], root: str) -> bool: return True +def sanctions_cluster(canonical: str, members: list[str], paths: list[str]) -> bool: + """True when this cluster line accounts for every instance of the group.""" + directories = set() + for path in paths: + if path != canonical and not any( + glob_matches(member, path) for member in members + ): + return False + directory = os.path.dirname(path) + if directory in directories: + return False + directories.add(directory) + return True + + +def sanctions(entry: Entry, instances: list[dict[str, Any]], root: str) -> bool: + if len(instances) < 2: + return False + paths = [relative(str(instance.get("file", "")), root) for instance in instances] + _, _, token, members = entry + if members: + return sanctions_cluster(token, members, paths) + return sanctions_plain(token, paths) + + def filter_document( document: dict[str, Any], - registries: list[tuple[str, list[tuple[int, str]]]], + registries: list[tuple[str, list[Entry]]], root: str, ) -> dict[str, Any]: kept: list[dict[str, Any]] = [] @@ -100,9 +147,9 @@ def filter_document( match = None if instances: for registry_path, entries in registries: - for number, entry in entries: + for entry in entries: if sanctions(entry, instances, root): - match = (registry_path, number, entry) + match = (registry_path, entry[0], entry[1]) break if match: break @@ -132,14 +179,19 @@ def floor_summary(document: dict[str, Any]) -> dict[str, Any]: that measured nothing keeps its "Measured nothing" headline instead of an unearned zero. """ + # A `partial` row measured what it did not skip, so its zero is earned too. measured = any( - row.get("measure") == "duplication" and row.get("status") == "ok" + row.get("measure") == "duplication" and row.get("status") in ("ok", "partial") for row in document.get("run") or [] ) if measured: summary = document.setdefault("summary", {}) summary.setdefault("duplicated_lines", 0) summary.setdefault("clone_groups", 0) + # The rollups are derived from surviving groups the same way, so a + # clone-free run states them as empty maps rather than omitting them. + summary.setdefault("by_lane", {}) + summary.setdefault("by_directory", {}) return document @@ -150,7 +202,7 @@ def main(argv: list[str]) -> int: parser.add_argument("--zero-floor", action="store_true") args = parser.parse_args(argv) - registries: list[tuple[str, list[tuple[int, str]]]] = [] + registries: list[tuple[str, list[Entry]]] = [] for path in args.registry: if not os.path.isfile(path): print(f"registry-filter.py: registry not found: {path}", file=sys.stderr) diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/test_cluster_clones.py b/plugins/code-metrics/skills/audit-duplication/scripts/test_cluster_clones.py new file mode 100755 index 0000000000..954931886b --- /dev/null +++ b/plugins/code-metrics/skills/audit-duplication/scripts/test_cluster_clones.py @@ -0,0 +1,235 @@ +#!/usr/bin/env python3 +"""Output-based tests for cluster-clones.py at its command line. + +The post-pass is a pure function from a document to a document, so every case +drives the script through subprocess with a small document on stdin (design +T13, the one seam per script). The committed captures +scripts/fixtures/tool-output/jscpd-aligned3.json and jscpd-offset3.json cover +the shapes jscpd itself reports for three copies; these cases cover the rule. +""" + +from __future__ import annotations + +import json +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +SCRIPT = Path(__file__).resolve().parent / "cluster-clones.py" + + +def instance(path: str, start: int, end: int) -> dict: + return {"file": path, "start_line": start, "end_line": end} + + +def pair( + first: tuple[str, int, int], + second: tuple[str, int, int], + lines: int = 41, + collector: str = "jscpd", + labels: list[str] | None = None, +) -> dict: + return { + "file": None, + "function": None, + "lane": "bash", + "instances": [instance(*first), instance(*second)], + "values": {"lines": lines, "tokens": 110}, + "collector": collector, + "labels": ["token-based"] if labels is None else labels, + } + + +def document(*rows: dict) -> dict: + return { + "schema": "code-metrics/v1", + "skill": "audit-duplication", + "measures": list(rows), + "excluded": [], + "summary": {"files": 0, "functions": 0, "over_reference": {}}, + } + + +def run(doc: dict | str, *args: str) -> subprocess.CompletedProcess: + return subprocess.run( + [sys.executable, str(SCRIPT), *args], + input=doc if isinstance(doc, str) else json.dumps(doc), + capture_output=True, + text=True, + check=False, + ) + + +def measures(result: subprocess.CompletedProcess) -> list[dict]: + return json.loads(result.stdout)["measures"] + + +A = "plugins/x/a/shared/shared-utils.sh" +B = "plugins/x/b/shared/shared-utils.sh" +C = "plugins/x/c/shared/shared-utils.sh" + + +class ClusterClonesTests(unittest.TestCase): + def test_three_aligned_copies_collapse_to_one_class(self) -> None: + # jscpd pairs every later copy with the first, so two rows name the + # same instance of the first copy. + result = run( + document(pair((A, 1, 41), (B, 1, 41)), pair((A, 1, 41), (C, 1, 41))) + ) + self.assertEqual(result.returncode, 0, result.stderr) + rows = measures(result) + self.assertEqual(len(rows), 1) + self.assertEqual([i["file"] for i in rows[0]["instances"]], [A, B, C]) + self.assertEqual(rows[0]["values"], {"lines": 41, "tokens": 110}) + self.assertEqual(rows[0]["labels"], ["token-based", "clustered"]) + + def test_the_lines_of_a_merged_class_count_once(self) -> None: + result = run( + document(pair((A, 1, 41), (B, 1, 41)), pair((A, 1, 41), (C, 1, 41))) + ) + rows = measures(result) + self.assertEqual(sum(row["values"]["lines"] for row in rows), 41) + + def test_overlap_without_an_identical_range_stays_two_groups(self) -> None: + # The third copy carries only part of the fragment, so its pair names + # the first copy with a shorter range. + result = run( + document( + pair(("c1.sh", 3, 36), ("c2.sh", 4, 37), lines=34), + pair(("c1.sh", 3, 26), ("c3.sh", 5, 28), lines=24), + ) + ) + rows = measures(result) + self.assertEqual(len(rows), 2) + self.assertTrue(all(len(row["instances"]) == 2 for row in rows)) + self.assertTrue(all("clustered" not in row["labels"] for row in rows)) + + def test_an_identical_range_with_different_lines_does_not_join(self) -> None: + result = run( + document( + pair((A, 1, 41), (B, 1, 41), lines=41), + pair((A, 1, 41), (C, 1, 41), lines=40), + ) + ) + self.assertEqual(len(measures(result)), 2) + + def test_a_three_instance_row_passes_through_unchanged(self) -> None: + row = pair((A, 1, 41), (B, 1, 41)) + row["instances"].append(instance(C, 1, 41)) + other = pair((A, 1, 41), ("plugins/x/d/shared/shared-utils.sh", 1, 41)) + result = run(document(row, other)) + rows = measures(result) + self.assertEqual(rows[0], row) + self.assertEqual(rows[1], other) + + def test_a_pair_from_another_collector_joins_on_an_identical_instance( + self, + ) -> None: + result = run( + document( + pair((A, 1, 41), (B, 1, 41)), + pair((A, 1, 41), (C, 1, 41), collector="cpd", labels=["cpd"]), + ) + ) + rows = measures(result) + self.assertEqual(len(rows), 1) + self.assertEqual(rows[0]["collector"], "jscpd") + self.assertEqual([i["file"] for i in rows[0]["instances"]], [A, B, C]) + + def test_merged_instances_are_sorted_by_path_then_start_line(self) -> None: + result = run( + document(pair((C, 1, 41), (B, 1, 41)), pair((A, 1, 41), (B, 1, 41))) + ) + rows = measures(result) + self.assertEqual(len(rows), 1) + self.assertEqual([i["file"] for i in rows[0]["instances"]], [A, B, C]) + + def test_a_chain_of_pairs_closes_into_one_class(self) -> None: + result = run( + document(pair((A, 1, 41), (B, 1, 41)), pair((B, 1, 41), (C, 1, 41))) + ) + rows = measures(result) + self.assertEqual(len(rows), 1) + self.assertEqual([i["file"] for i in rows[0]["instances"]], [A, B, C]) + + def test_the_class_is_emitted_where_its_first_pair_stood(self) -> None: + unrelated = pair(("y.sh", 1, 10), ("z.sh", 1, 10), lines=10) + result = run( + document( + pair((A, 1, 41), (B, 1, 41)), unrelated, pair((A, 1, 41), (C, 1, 41)) + ) + ) + rows = measures(result) + self.assertEqual(len(rows), 2) + self.assertEqual(len(rows[0]["instances"]), 3) + self.assertEqual(rows[1], unrelated) + + def test_rows_without_instances_and_the_summary_pass_through(self) -> None: + file_row = { + "file": "a.sh", + "function": None, + "lane": "bash", + "values": {"lines": 3}, + "collector": "scc", + } + doc = document(file_row, pair((A, 1, 41), (B, 1, 41))) + doc["summary"] = {"files": 9, "functions": 0, "over_reference": {}} + result = run(doc) + out = json.loads(result.stdout) + self.assertEqual(out["measures"][0], file_row) + self.assertEqual(out["summary"], doc["summary"]) + self.assertEqual(out["excluded"], []) + + def test_stdin_that_is_not_json_is_a_usage_error(self) -> None: + result = run("not json") + self.assertEqual(result.returncode, 2) + self.assertIn("not a JSON document", result.stderr) + + def test_an_unknown_argument_is_a_usage_error(self) -> None: + self.assertEqual(run(document(), "--root").returncode, 2) + self.assertEqual(run(document(), "--registry", "x").returncode, 2) + + def test_root_sorts_instances_by_their_root_relative_path(self) -> None: + # From `lib/`, the dispatcher names the root copy `hook-utils.sh` and + # a plugin copy `../plugins/a/hooks/hook-utils.sh`, which sorts first + # as text; relative to the root the `lib/` copy comes first, so the + # rollup attributes the class the same way a root run does. + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + lib = root / "lib" + lib.mkdir() + doc = document( + pair( + ("../plugins/a/hooks/hook-utils.sh", 1, 41), + ("hook-utils.sh", 1, 41), + ), + pair( + ("../plugins/b/hooks/hook-utils.sh", 1, 41), + ("hook-utils.sh", 1, 41), + ), + ) + result = subprocess.run( + [sys.executable, str(SCRIPT), "--root", tmp], + input=json.dumps(doc), + capture_output=True, + text=True, + cwd=lib, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + rows = measures(result) + self.assertEqual(len(rows), 1) + self.assertEqual( + [i["file"] for i in rows[0]["instances"]], + [ + "hook-utils.sh", + "../plugins/a/hooks/hook-utils.sh", + "../plugins/b/hooks/hook-utils.sh", + ], + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py b/plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py index f143a6f39f..cc1025ce69 100755 --- a/plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py +++ b/plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py @@ -147,6 +147,114 @@ def test_the_committed_cluster_registry_declares_the_fixture_path(self) -> None: self.assertEqual(out["measures"], []) self.assertEqual(out["excluded"][0]["path"], "shared/shared-utils.sh") + def write_registry(self, text: str) -> Path: + path = Path(self.tmp.name) / "clusters.txt" + path.write_text(text, encoding="utf-8") + return path + + def test_a_cluster_line_excludes_the_canonical_plus_its_copies(self) -> None: + registry = self.write_registry( + "# canonical and copies\nlib/hook-utils.sh -> plugins/*/hooks/hook-utils.sh\n" + ) + doc = document( + [ + "lib/hook-utils.sh", + "plugins/one/hooks/hook-utils.sh", + "plugins/two/hooks/hook-utils.sh", + ] + ) + result = run(doc, "--root", ".", "--registry", str(registry)) + self.assertEqual(result.returncode, 0, result.stderr) + out = json.loads(result.stdout) + self.assertEqual(out["measures"], []) + entry = out["excluded"][0] + self.assertEqual( + entry["path"], "lib/hook-utils.sh -> plugins/*/hooks/hook-utils.sh" + ) + self.assertEqual(entry["line"], 2) + self.assertEqual(len(entry["instances"]), 3) + + def test_a_glob_member_matches_and_a_stranger_keeps_the_group(self) -> None: + registry = self.write_registry( + "lib/parse.sh -> plugins/*/skills/*/scripts/parse.sh " + "plugins/*/skills/*/scripts/lib/parse.sh\n" + ) + sanctioned = document( + [ + "lib/parse.sh", + "plugins/a/skills/x/scripts/parse.sh", + "plugins/b/skills/y/scripts/lib/parse.sh", + ] + ) + result = run(sanctioned, "--root", ".", "--registry", str(registry)) + self.assertEqual(json.loads(result.stdout)["measures"], []) + stranger = document(["lib/parse.sh", "plugins/a/hooks/parse.sh"]) + result = run(stranger, "--root", ".", "--registry", str(registry)) + out = json.loads(result.stdout) + self.assertEqual(len(out["measures"]), 1) + self.assertEqual(out["excluded"], []) + + def test_two_cluster_instances_in_one_directory_keep_the_group(self) -> None: + registry = self.write_registry("lib/a.sh -> plugins/*/hooks/*.sh\n") + doc = document(["lib/a.sh", "plugins/one/hooks/a.sh", "plugins/one/hooks/b.sh"]) + result = run(doc, "--root", ".", "--registry", str(registry)) + out = json.loads(result.stdout) + self.assertEqual(len(out["measures"]), 1) + self.assertEqual(out["excluded"], []) + + def test_a_plain_line_with_a_space_is_one_path(self) -> None: + registry = self.write_registry("hooks/shared file.sh\n") + doc = document( + ["plugins/one/hooks/shared file.sh", "plugins/two/hooks/shared file.sh"] + ) + result = run(doc, "--root", ".", "--registry", str(registry)) + out = json.loads(result.stdout) + self.assertEqual(out["measures"], []) + self.assertEqual(out["excluded"][0]["path"], "hooks/shared file.sh") + + def test_the_first_matching_line_in_file_order_wins(self) -> None: + registry = self.write_registry("lib/x.sh -> plugins/*/hooks/x.sh\nhooks/x.sh\n") + doc = document(["plugins/one/hooks/x.sh", "plugins/two/hooks/x.sh"]) + result = run(doc, "--root", ".", "--registry", str(registry)) + entry = json.loads(result.stdout)["excluded"][0] + self.assertEqual( + (entry["path"], entry["line"]), ("lib/x.sh -> plugins/*/hooks/x.sh", 1) + ) + reversed_registry = self.write_registry( + "hooks/x.sh\nlib/x.sh -> plugins/*/hooks/x.sh\n" + ) + result = run(doc, "--root", ".", "--registry", str(reversed_registry)) + entry = json.loads(result.stdout)["excluded"][0] + self.assertEqual((entry["path"], entry["line"]), ("hooks/x.sh", 1)) + + def test_cwd_relative_instances_from_a_subdirectory_still_match(self) -> None: + root = Path(self.tmp.name) + sub = root / "plugins" / "code-metrics" + sub.mkdir(parents=True) + registry = self.write_registry( + "lib/hook-utils.sh -> plugins/*/hooks/hook-utils.sh\n" + ) + doc = document(["../../lib/hook-utils.sh", "../one/hooks/hook-utils.sh"]) + result = subprocess.run( + [ + sys.executable, + str(SCRIPT), + "--root", + str(root), + "--registry", + str(registry), + ], + input=json.dumps(doc), + capture_output=True, + text=True, + cwd=sub, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + out = json.loads(result.stdout) + self.assertEqual(out["measures"], []) + self.assertEqual(len(out["excluded"]), 1) + def test_the_zero_floor_states_zero_once_a_collector_ran(self) -> None: doc = document() doc["run"] = [{"lane": "bash", "measure": "duplication", "status": "ok"}] @@ -156,6 +264,30 @@ def test_the_zero_floor_states_zero_once_a_collector_ran(self) -> None: summary = json.loads(result.stdout)["summary"] self.assertEqual((summary["duplicated_lines"], summary["clone_groups"]), (0, 0)) + def test_the_zero_floor_states_empty_rollup_maps(self) -> None: + doc = document() + doc["run"] = [{"lane": "bash", "measure": "duplication", "status": "ok"}] + doc["summary"] = {"files": 0, "functions": 0, "over_reference": {}} + result = run(doc, "--root", ".", "--zero-floor") + summary = json.loads(result.stdout)["summary"] + self.assertEqual((summary["by_lane"], summary["by_directory"]), ({}, {})) + + def test_the_zero_floor_counts_a_partial_lane_as_measured(self) -> None: + doc = document() + doc["run"] = [ + { + "lane": "bash", + "measure": "duplication", + "status": "partial", + "reason": "1 of 3 files skipped by duplication.max_size 1mb / max_lines none", + } + ] + doc["summary"] = {"files": 0, "functions": 0, "over_reference": {}} + result = run(doc, "--root", ".", "--zero-floor") + self.assertEqual(result.returncode, 0, result.stderr) + summary = json.loads(result.stdout)["summary"] + self.assertEqual((summary["duplicated_lines"], summary["clone_groups"]), (0, 0)) + def test_the_zero_floor_leaves_a_run_that_measured_nothing_alone(self) -> None: doc = document() doc["run"] = [ diff --git a/plugins/code-metrics/skills/setup/templates/config-template.yaml b/plugins/code-metrics/skills/setup/templates/config-template.yaml index 4722cc6e4f..a5096c8374 100644 --- a/plugins/code-metrics/skills/setup/templates/config-template.yaml +++ b/plugins/code-metrics/skills/setup/templates/config-template.yaml @@ -29,6 +29,9 @@ duplication: min_lines: 5 ignore: [] registries: [] # older name for scope.registries; read only when the scope-level list is empty + max_lines: null # files with more lines are skipped and reported; null (or 0) means no line cap + max_size: 1mb # files larger than this are skipped and reported; jscpd 5's own guard, SonarJS uses 1000kb + rollup_depth: 2 # per-directory rollup rows listed to this depth in the markdown report coverage: artifacts: [] # explicit artifact paths; empty means auto-discover diff --git a/scripts/check-cross-plugin-source-drift.sh b/scripts/check-cross-plugin-source-drift.sh index 0b2fa70cf3..cc66ba41b3 100755 --- a/scripts/check-cross-plugin-source-drift.sh +++ b/scripts/check-cross-plugin-source-drift.sh @@ -158,6 +158,19 @@ if [[ -f "$registry" ]]; then # owns the two comment families and why they must stay distinct). # shellcheck disable=SC2310 # the non-zero return IS the handled case; the library reports it read_list::into registry_entries "$registry" --comments inline || exit 2 + # A line containing ` -> ` is a cluster line (` -> ...`, + # a root-relative canonical copy and the plugin paths or globs that carry + # it). It belongs to the duplication audit's reader + # (plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py), + # which excludes the whole class from its clone count; this check keys + # clusters by path-within-plugin, so the line is dropped from the entry list + # here and is neither registered nor reported stale. + path_entries=() + for line in ${registry_entries[@]+"${registry_entries[@]}"}; do + [[ "$line" == *" -> "* ]] && continue + path_entries+=("$line") + done + registry_entries=(${path_entries[@]+"${path_entries[@]}"}) for line in ${registry_entries[@]+"${registry_entries[@]}"}; do registered["$line"]=1 done diff --git a/scripts/check-cross-plugin-source-drift.test.sh b/scripts/check-cross-plugin-source-drift.test.sh index 6b3efc1fd7..b465bb8216 100755 --- a/scripts/check-cross-plugin-source-drift.test.sh +++ b/scripts/check-cross-plugin-source-drift.test.sh @@ -215,6 +215,37 @@ else fi rm -rf "$f" +# --- a cluster line (` -> `) is the duplication audit's, not this check's --- +# +# ` -> ...` names a root-relative canonical copy and the +# plugin paths or globs that carry it, for registry-filter.py in the +# code-metrics plugin. This check keys clusters by path-within-plugin, so the +# line must be skipped: registering it would report it REGISTRY STALE on every +# run, since no plugin carries a path spelled `lib/... -> ...`. +f="$(new_fixture)" +plugin_file "$f" alpha hooks/shared.sh "same" +plugin_file "$f" beta hooks/shared.sh "same" +registry "$f" "hooks/shared.sh" "lib/shared.sh -> plugins/*/hooks/shared.sh" +if out="$(run_check "$f" 2>&1)"; then + if grep -q 'REGISTRY STALE' <<<"$out"; then + fail "a cluster line must not be reported stale, got: $out" + else + ok "--check skips a cluster line instead of registering it" + fi +else + fail "--check should pass with a cluster line beside a registered path, got: $out" +fi +if out="$(run_discover "$f" 2>&1)"; then + if grep -q ' -> ' <<<"$out"; then + fail "discover must not list a cluster line, got: $out" + else + ok "discover leaves a cluster line out of the inventory" + fi +else + fail "discover should exit 0 with a cluster line in the registry, got: $out" +fi +rm -rf "$f" + # --- production registry: every cluster documents its enforcement path (#2404) - REGISTRY="$SELF_DIR/cross-plugin-source-registry.txt" if [[ ! -f "$REGISTRY" ]]; then diff --git a/scripts/cross-plugin-source-registry.txt b/scripts/cross-plugin-source-registry.txt index 6feb37c66a..cfe0c77ba0 100644 --- a/scripts/cross-plugin-source-registry.txt +++ b/scripts/cross-plugin-source-registry.txt @@ -9,14 +9,26 @@ # (already a required CI job) and names its canonical copy instead, so a drift # failure says which direction to fix. # -# One path-within-plugin per line, relative to each plugin's own root. +# One path-within-plugin per line, relative to each plugin's own root. A line +# of the form ` -> ...` is a cluster line: the root-relative +# canonical copy, then the plugin paths or globs that carry it. It is read only +# by the code-metrics duplication audit (registry-filter.py), which excludes +# the canonical plus every copy as one sanctioned class; this script skips it. # Dedicated check: scripts/sync-hook-utils.sh --check (CI: hook-utils-sync) hooks/hook-utils.sh +# Dedicated check: scripts/sync-hook-utils.sh --check (CI: hook-utils-sync). +# Cluster line for the duplication audit: the root canonical and the copies. +lib/hook-utils.sh -> plugins/*/hooks/hook-utils.sh + # Dedicated check: scripts/sync-rewrite-guard.sh --check (CI: rewrite-guard-sync) hooks/rewrite-guard.sh +# Dedicated check: scripts/sync-rewrite-guard.sh --check (CI: rewrite-guard-sync). +# Cluster line for the duplication audit: the root canonical and the copies. +lib/rewrite-guard.sh -> plugins/*/hooks/rewrite-guard.sh + # Dedicated check: scripts/validate-plugin-contracts.mjs (lifecycleProtocolCopies) reference/artifact-protocol.md @@ -36,6 +48,19 @@ lib/state-key.sh # second plugin consumes it. # scripts/index-regen.sh +# Dedicated check: scripts/sync-index-regen.sh --check. Cluster line for the +# duplication audit: the root canonical and its copy are one class today, so +# the line is live even while the path-within-plugin entry above stays out. +lib/index-regen.sh -> plugins/*/scripts/index-regen.sh + +# Dedicated check: scripts/sync-resolve-convention-pattern.sh --check. +# Cluster line for the duplication audit: the root canonical and its copy. +lib/resolve-convention-pattern.sh -> plugins/*/hooks/resolve-convention-pattern.sh + +# Dedicated check: scripts/sync-parse-concern-value.sh --check. Cluster line +# for the duplication audit: the copies sit at two shapes within a plugin. +lib/parse-concern-value.sh -> plugins/*/skills/*/scripts/parse-concern-value.sh plugins/*/skills/*/scripts/lib/parse-concern-value.sh + # Dedicated check: scripts/sync-spawn-noise.sh --check (CI: spawn-noise-sync) lib/spawn_noise.py