From 9d708a46b24258cb34c2214fe95e2bf9d9b84e79 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 03:05:25 +0000 Subject: [PATCH 01/15] docs(code-metrics): lock the audit-duplication fixes brief Interview contract for the six audit-duplication gaps a whole-tree run exposed: size-cap skip detection and defaults, pair-to-class merging, a registry grammar for canonical copies outside a plugin, report sort and rollups with additive JSON fields, the duplication summary line, and the no-detector experience. Brief only; the Plan section is filled next. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01QShQwS5yDYYToBmofdEY5N --- .../code-metrics-duplication-audit/PLAN.md | 141 ++++++++++++++++++ 1 file changed, 141 insertions(+) create mode 100644 docs/topics/code-metrics-duplication-audit/PLAN.md diff --git a/docs/topics/code-metrics-duplication-audit/PLAN.md b/docs/topics/code-metrics-duplication-audit/PLAN.md new file mode 100644 index 0000000000..7d2fb22701 --- /dev/null +++ b/docs/topics/code-metrics-duplication-audit/PLAN.md @@ -0,0 +1,141 @@ +# Plan: code-metrics audit-duplication fixes + +Issue: to be filed once the Brief is confirmed; this line then reads `Closes #`. + +## Brief + +### TLDR + +- `audit-duplication` reports skipped files instead of hiding them: the jscpd adapter pre-filters + by byte size and line count, passes both caps explicitly on 4.x and 5.x, and marks the lane + `partial` when anything was skipped; defaults are no line cap and a 1mb byte cap. +- jscpd's pair reports are merged into clone classes before summarizing, so N copies count once. +- The sanctioned-replication registry gains multi-token lines (` ...`) + so a canonical file outside any plugin can declare its copies; this repo's drift checker skips + those lines and six such lines are added here. +- The markdown report sorts clone rows by duplicated lines, adds per-lane and per-directory rollups + (cumulative, counts beside share, computed after exclusion, listed to depth 2), and the JSON gains + `summary.by_lane` / `summary.by_directory` as additive `v1` fields with an explicit ignore-unknown + rule in the schema reference. +- The duplication summary line drops "Functions" and "Over reference"; a run with no detector prints + one consolidated install headline and the skill offers, never performs, the install. Version 0.1.9. + +### Goal + +A whole-tree or change-scoped duplication audit on any repository, this one included, produces +numbers a reader can act on without re-aggregating: no file is silently dropped by a size cap, a +fragment copied N times is one group counted once, replication the repository declares about itself +(including copies of a root-level canonical file) is excluded and shown as an exclusion, and the +report says where the surviving duplication sits by lane and by directory. On this repository the +audit reads clean apart from genuine duplication. + +### Constraints + +- The plugin never installs, downloads, or `npx`-fetches a detector; SKILL.md may instruct Claude to + offer the install command and run it only on the user's confirmation. +- The report emits no finding, severity, or exit-code gate; duplication has no reference value. +- The `code-metrics/v1` schema string is unchanged; every JSON change is additive (new optional + fields), and `reference/report-schema.md` states that readers ignore unknown keys. +- Single-token registry lines keep their exact meaning; the drift checker + (`scripts/check-cross-plugin-source-drift.sh`) keeps its behavior for them and skips multi-token + lines. +- Both jscpd 4.x (`latest-4` = 4.3.0) and 5.x (5.2.0) stay supported by the adapter; the adapter + always passes explicit `--max-lines` and `--max-size` on 4.x (its `0` means "default" for lines and + "skip all" for size) and never emits `--max-size 0`. +- An unmeasured value is `null`, never `0`; a run that measured nothing keeps "Measured nothing". +- Validate with `scripts/affected-tests.sh --run`; every changed file maps to at least one suite. +- No `lib/hook-utils.sh` or other cross-plugin synced source is edited; registry lines mirror the + `src=` each `scripts/sync-*.sh` already declares. +- One issue, one draft PR whose body opens with `Closes #` and carries the four required + sections; CHANGELOG entry under `[0.1.9]`. + +### Acceptance criteria + +- Running `audit-duplication.sh --all` on this repository with jscpd 4.3.0 and again with 5.2.0 + reports identical clone groups for files under 1mb, and the run table's bash row reads `partial` + naming the count and largest of any files the adapter's pre-filter skipped; with the shipped + defaults no file in this repository is skipped. +- `duplication.max_lines` defaults to `null` (no cap) and `duplication.max_size` to `1mb`; both are + documented in `reference/config.md` (gated against `config-defaults.json`) and exported to the + adapter, and the adapter's tests cover the 4.x explicit-cap translation and the never-`0` rule. +- IF every file in a lane is skipped by the caps, THEN that lane's run row reads `partial` with the + reason, no collector is invoked for it, and the script never exits 3 for that cause. +- Seventeen identical copies of `hook-utils.sh` (root `lib/` plus sixteen plugins) produce exactly + one clone group with seventeen instances, and with the registry line + `lib/hook-utils.sh plugins/*/hooks/hook-utils.sh` that group appears once under `excluded[]` + with `duplicated_lines` counted once, not sixteen times. +- Two groups whose instances overlap without identical line ranges (the `hook-telemetry-sink.sh` + shape) stay separate groups after the merge. +- A multi-token registry line excludes a group only when every instance matches the canonical path + or one of the copy paths or globs and the instances sit in distinct carrying directories; two copies + inside one directory still count as duplication; a single-token line behaves exactly as before + (existing registry tests unchanged and passing). +- `scripts/check-cross-plugin-source-drift.sh --check` passes on this repository with the six new + multi-token lines present, and its tests cover a multi-token line being skipped. +- After this change, `audit-duplication.sh --registry scripts/cross-plugin-source-registry.txt --all` + on this repository reports zero surviving groups whose instances include a file under root `lib/` + or `.claude/hooks/`. +- WHILE no registry is configured and none is passed, the report's `excluded[]` is empty and the + summary line says so. +- The markdown Measures table for a duplication document lists clone groups in descending order of + duplicated lines, and the report carries a `## Rollup` section with a per-lane table and a + per-directory table (rows to depth 2 by default, `duplication.rollup_depth` configurable) whose + numbers are cumulative up the tree, carry `groups` and `duplicated_lines`, and sum consistently + with `summary.duplicated_lines` after exclusion. +- The JSON `summary` carries `by_lane` and `by_directory` with the same numbers; `schema` is still + `code-metrics/v1`; `reference/report-schema.md` documents both fields and states that readers + ignore unknown keys; `verification:measure`'s consumption is unaffected. +- The duplication summary line reads `Files with clones: N.` followed by the duplicated-lines and + exclusion lines, with no "Functions" or "Over reference" text; the sibling skills' summary lines are + byte-identical to today's. +- When no duplication collector resolves, the markdown opens with one headline naming the install + command and `/code-metrics:setup`, the per-lane rows no longer repeat the install hint, and + SKILL.md instructs Claude to offer the install and run it only on confirmation. +- The jscpd adapter's docstring and `reference/collectors.md` state that 4.x and 5.x are both + translated, pin 5.2.0, and note the `kind` field; the schema reference names "intentional clones" + beside "sanctioned replication". +- `plugin.json` reads 0.1.9 and `CHANGELOG.md` carries a `[0.1.9]` entry covering every item above; + `scripts/affected-tests.sh --run` passes. + +### Captured assumptions + +- Version bumps to 0.1.9, not 0.2.0, because this plugin's changelog bumps patch for features + (0.1.7 was a `feat`). Revisit if the marketplace's release convention says a new registry grammar + or JSON fields require a minor bump. +- Once-per-class counting follows PMD CPD and SonarQube; the fetched literature is silent on how + duplicated lines are totalled, and jscpd v5 is the one tool that sums per pair. Revisit if a + standard sets a duplicated-lines definition. +- The merge keys on identical (file, start_line, end_line) instances and equal `lines`; closure is + exact only for type-1/type-2 clones, which is all the adapter receives today because it passes no + `--max-gap-lines`. Revisit if `similar` clones are ever enabled. +- `rollup_depth` default 2 is the plugin's choice; no upstream sets a depth (SonarQube and Codacy + roll up every directory). Revisit if a consuming repository's layout makes depth 2 meaningless. +- The byte cap of 1mb aligns with jscpd 5.0.7's parser guard and SonarJS's 1000kb generated-code + rule; no line cap by default aligns with jscpd 5, PMD CPD, SonarQube, and Linguist. Revisit if + jscpd changes its default in a later major. +- The cluster lines live in the existing registry file because the plugin's own documentation names + that file as the shape; this is a repository-convention choice with no external authority. + Revisit if the drift checker grows a second consumer of the file. +- "Sanctioned replication" stays the plugin's term; the literature's term "intentional clones" + (Cordy 2008) is named beside it. No upstream tool models the canonical-plus-copies relation, so + no vocabulary conflict exists. +- The upstream doc drift found in jscpd (`docs/rust.md` "no limit" for `--max-size`, "per block" + help text for `--max-lines`, stale jscpd.dev v5 defaults) is reported separately, not here. + +### Out-of-scope + +- Installing a detector on the user's behalf, or an `npx` fallback. +- A pass/fail gate, threshold, or severity for duplication. +- Cross-language clone detection. +- Changing `dupl` or `cpd` adapters beyond passing their rows through the new merge and rollup + steps unchanged. +- A second registry file or a manifest generated from the sync scripts. +- Migrating the registry's single-token lines to the new grammar. + +### Deferred questions + +- None. Every question registered in the interview was answered; no row was deferred or blocked. + +## Plan + + From 33bbc0e980c4e098f02f1fefd1dfd491bed0a9d7 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 03:09:10 +0000 Subject: [PATCH 02/15] docs(code-metrics): link the audit-duplication brief to issue #4068 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01QShQwS5yDYYToBmofdEY5N --- docs/topics/code-metrics-duplication-audit/PLAN.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/code-metrics-duplication-audit/PLAN.md b/docs/topics/code-metrics-duplication-audit/PLAN.md index 7d2fb22701..e09be25b30 100644 --- a/docs/topics/code-metrics-duplication-audit/PLAN.md +++ b/docs/topics/code-metrics-duplication-audit/PLAN.md @@ -1,6 +1,6 @@ # Plan: code-metrics audit-duplication fixes -Issue: to be filed once the Brief is confirmed; this line then reads `Closes #`. +Closes melodic-software/claude-code-plugins#4068. ## Brief From 144d25d0f836c11a89e63e5bc58461eb5092c618 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 03:16:10 +0000 Subject: [PATCH 03/15] docs(code-metrics): draft the audit-duplication implementation plan Five phases under the locked Brief (caps end to end, pair-to-class merge, registry cluster lines with the drift-checker skip, report rollups and summary line, docs and release), plus the Tier B design early-exit with the type sketch the phases build to. Draft pending the fresh-context plan review and the stress-test; approval comes after both. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01QShQwS5yDYYToBmofdEY5N --- .../code-metrics-duplication-audit/PLAN.md | 312 +++++++++++++++++- .../design/design-resolution.md | 86 +++++ 2 files changed, 397 insertions(+), 1 deletion(-) create mode 100644 docs/topics/code-metrics-duplication-audit/design/design-resolution.md diff --git a/docs/topics/code-metrics-duplication-audit/PLAN.md b/docs/topics/code-metrics-duplication-audit/PLAN.md index e09be25b30..d58d6b40f9 100644 --- a/docs/topics/code-metrics-duplication-audit/PLAN.md +++ b/docs/topics/code-metrics-duplication-audit/PLAN.md @@ -138,4 +138,314 @@ audit reads clean apart from genuine duplication. ## Plan - +### Goal + +**What**: the six audit-duplication fixes the Brief locks, in `plugins/code-metrics` at 0.1.9, +plus one drift-checker change and six registry lines in this repository, in one draft PR that +closes issue 4068. + +**Why**: the whole-tree audit hid skipped files behind `complete`, inflated every count by pair +reporting, could not declare this repo's own canonical copies, and left the reader to aggregate +by hand; each number the skill reports has to be one a reader can act on. + +### Standards grounding + +No `.claude/standards.yaml` and no `docs/standards/README.md` exist, so the ladder's rung 4 +applied: inferred from repository conventions not auto-loaded. Offer stands to bootstrap an index +through the planning setup; nothing was written. + +| Surface | Sections cited | Layer provenance | +|---|---|---| +| Python | `.claude/rules/ruff-pin.md` (lint only through `scripts/run-ruff.sh check`) | team, path-scoped | +| Skill body | `.claude/rules/skill-bodies-state-current-rules.md` (current rule and reason, no incident narration, `## Next` before `## Gotchas`); `.claude/rules/vendor-docs-are-not-style.md` (house style, `/ai-slop:audit`) | team, path-scoped | +| Shell tests | `docs/conventions/shell-test-helpers/README.md` (per-plugin `pass`/`fail` helpers stay per plugin; repo-tooling suites use `scripts/lib/test-harness.sh`) | team | +| Upstream facts | `docs/conventions/upstream-drift/README.md` (a restated upstream specific carries claim, basis, as-of date, recheck trigger) | team | +| Validation | `AGENTS.md` "Validate a change" (`scripts/affected-tests.sh --run`; a changed file mapping to zero suites is an error) | team, ambient | +| Release | `scripts/check-changelog-parity.sh --check-bump` (a manifest bump must add a `## []` entry) | team | + +### Approach + +Five phases, integration slice first. Phase 1 lands the cap machinery end to end (config key to +run row) because it is the only phase that changes what jscpd is asked to scan, and everything +after it measures the same scope. Phase 2 and Phase 3 are file-disjoint and together satisfy the +Brief's headline criterion (one 17-instance `hook-utils.sh` group, excluded once). Phase 4 is the +rendering layer over the shape Phases 2 and 3 produce. Phase 5 is docs and release. + +Build technique: kept tracer-bullet slice (Phase 1 runs `audit-duplication.sh --all` on this +repository under both jscpd majors as its runtime probe); no throwaway spike, since the research +already reproduced every tool behavior the design relies on. + +### Phase 1: Explicit caps, adapter pre-filter, partial run row [TODO] + +Review: code-design + +1. Add `duplication.max_lines` (`null`), `duplication.max_size` (`"1mb"`), and + `duplication.rollup_depth` (`2`) to `scripts/config-defaults.json`; add the three rows to + `reference/config.md` (the gate `scripts/check-code-metrics-config-reference.py` pins key set + and default rendering); add the three keys to `skills/setup/templates/config-template.yaml` + (pinned leaf for leaf by `test_setup_apply.py`). +2. `audit-duplication.sh`: read the two caps beside the existing tunables and export + `CODE_METRICS_DUP_MAX_LINES` (empty when null) and `CODE_METRICS_DUP_MAX_SIZE`; keep the + existing three exports byte-identical. +3. `dispatch.sh`: before each `collect`, export `CODE_METRICS_RUN_NOTE_FILE` pointing at + `$WORK/note.$lane.$measure.$tool`; after a `collect` that exits 0, if that file is non-empty, + write the run row as `partial` with the file's single line as the reason, else `ok` as today. + [EXEC-SHAPE] File-per-channel matches how `dispatch.sh` already isolates each adapter's stdout + and stderr into `$WORK` files. +4. `collectors/jscpd.py`: parse `CODE_METRICS_DUP_MAX_SIZE` with jscpd's own unit grammar + (`kb`/`mb`/raw bytes) and `CODE_METRICS_DUP_MAX_LINES`; pre-filter the file list by `os.stat` + size and newline count; pass `--max-size ` always and `--max-lines ` + always (4.x reads `0` as "use default", so the null cap is spelled as a large explicit number + [EXEC-SHAPE]); never emit `--max-size 0`; when files were skipped, write one line to the note + file (`N of M files skipped by duplication.max_size / max_lines ; largest: + ()`); when the pre-filter leaves zero files, write the note and return 0 without invoking + jscpd (4.x would write no report and the current exit-3 path would call that a failure). Extend + the `probe` docstring and module docstring: 4.x and 5.x both translate; drop the "jscpd 4 is not + translated" sentence; record the 5.2.0 `kind` field. +5. `reference/collectors.md`: jscpd row pinned to 5.2.0 with a fresh as-of date and the recheck + trigger unchanged; add the 4.x maintenance-line fact (`latest-4` = 4.3.0) as a four-part + verification record. + +**Files Affected** + +| File | Action | What changes | +|---|---|---| +| `plugins/code-metrics/scripts/config-defaults.json` | Modify | three keys under `duplication` | +| `plugins/code-metrics/reference/config.md` | Modify | three key rows | +| `plugins/code-metrics/skills/setup/templates/config-template.yaml` | Modify | three keys | +| `plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh` | Modify | two exports | +| `plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh` | Modify | export assertions | +| `plugins/code-metrics/scripts/dispatch.sh` | Modify | note-file channel, `partial` row | +| `plugins/code-metrics/scripts/dispatch.test.sh` | Modify | partial-row case via the real jscpd adapter and a stub binary | +| `plugins/code-metrics/scripts/collectors/jscpd.py` | Modify | caps, pre-filter, note file, docstring | +| `plugins/code-metrics/scripts/collectors/test_jscpd.py` | Modify | argv, skip, all-skipped, size-grammar cases | +| `plugins/code-metrics/reference/collectors.md` | Modify | jscpd row, 4.x record | + +**Sanity Check:** + +- `python3 scripts/check-code-metrics-config-reference.py` exits 0. +- `python3 -m unittest plugins/code-metrics/scripts/collectors/test_jscpd.py` exits 0 and its + argv-log case asserts `--max-size 1mb` and `--max-lines 1000000` present and `--max-size 0` absent. +- `bash plugins/code-metrics/scripts/dispatch.test.sh` exits 0 with a case whose JSON has + `run[].status == "partial"` and a reason matching `^[0-9]+ of [0-9]+ files skipped`. +- Runtime probe: with jscpd 5.2.0 on PATH, + `audit-duplication.sh --json --registry scripts/cross-plugin-source-registry.txt --all | jq '.run[]|select(.lane=="bash")|.status'` + prints `"ok"` (no file in this repo exceeds 1mb), and the same command with jscpd 4.3.0 on PATH + prints `"ok"` with `summary.files` equal between the two runs. + +### Phase 2: Merge pairs into clone classes [TODO] + +Review: code-design + +1. New `skills/audit-duplication/scripts/cluster-clones.py` (stdin document, stdout document): + union-find over rows carrying exactly two `instances[]` whose `collector` reports pairs (rows + with three or more instances pass through untouched, which covers `dupl` and `cpd`); two rows + join when they share an instance with identical `(file, start_line, end_line)` and equal + `values.lines`; the merged row keeps the first row's `values`, the union of instances in + first-seen order, and appends `clustered` to `labels`. Rows without `instances` pass through. + Exit 0 on a printed document, 2 on a non-JSON stdin. +2. `audit-duplication.sh`: pipe `report.json` through `cluster-clones.py` before `registry-filter.py`. +3. Fixture: add `scripts/fixtures/sources/cluster/gamma/shared/shared-utils.sh` (byte-identical + third copy) and a committed capture `scripts/fixtures/tool-output/jscpd-three.json` produced by + a real jscpd 5.2.0 run over the three copies (two pair rows against `alpha`); the existing + two-copy capture and every test that replays it stay unchanged. +4. `test_cluster_clones.py`: three copies collapse to one three-instance group counted once; + overlapping-but-not-identical ranges stay separate; a three-instance input row passes through; + `summary` is left to `report.py resummarize`. + +**Files Affected** + +| File | Action | What changes | +|---|---|---| +| `plugins/code-metrics/skills/audit-duplication/scripts/cluster-clones.py` | Create | the post-pass | +| `plugins/code-metrics/skills/audit-duplication/scripts/test_cluster_clones.py` | Create | output-based tests | +| `plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh` | Modify | pipeline step | +| `plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh` | Modify | three-copy case through the stub | +| `plugins/code-metrics/scripts/fixtures/sources/cluster/gamma/shared/shared-utils.sh` | Create | third copy | +| `plugins/code-metrics/scripts/fixtures/tool-output/jscpd-three.json` | Create | capture | + +**Sanity Check:** + +- `python3 -m unittest plugins/code-metrics/skills/audit-duplication/scripts/test_cluster_clones.py` exits 0. +- `bash plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh` exits 0 + with a case asserting `summary.clone_groups == 1` and `len(measures[0].instances) == 3` on the + three-copy capture with no registry. +- `cmp` the three fixture copies: identical. + +### Phase 3: Registry cluster lines, drift checker, this repo's six lines [TODO] + +Review: code-design + +1. `registry-filter.py`: `read_registry` returns, per line, either a single token or a + `(canonical, [members...])` tuple; `sanctions()` for a cluster line matches each instance's + repo-relative path against the canonical path or any member via `pathglob.matches`, requires + distinct carrying directories (the prefix in front of the matched token; the canonical path's + parent for itself), and records `path` as the line's text. Single-token behavior byte-identical. + Pre-flight (done in planning): the only other parser of the registry file is + `scripts/check-cross-plugin-source-drift.sh`; `check-shell-portability.sh` compares whole lines + to a path-within-plugin and cannot match a multi-token line; every other mention is a comment. +2. `scripts/check-cross-plugin-source-drift.sh`: in the registry load loop, `continue` on a line + containing whitespace after trimming, with a comment naming the cluster-line grammar and its + owner (the code-metrics registry filter). `check-cross-plugin-source-drift.test.sh`: a case + where a multi-token line neither registers nor reports `REGISTRY STALE`. +3. `scripts/cross-plugin-source-registry.txt`: a commented section "Canonical sources outside a + plugin (cluster lines; read by code-metrics audit-duplication, skipped by the drift checker)" + with six lines, each mirroring its sync script's `src=` and copy paths: + `lib/hook-utils.sh plugins/*/hooks/hook-utils.sh`; + `lib/rewrite-guard.sh plugins/*/hooks/rewrite-guard.sh`; + `lib/index-regen.sh plugins/*/scripts/index-regen.sh`; + `lib/resolve-convention-pattern.sh plugins/*/hooks/resolve-convention-pattern.sh`; + `lib/parse-concern-value.sh plugins/*/skills/*/scripts/parse-concern-value.sh plugins/*/skills/*/scripts/lib/parse-concern-value.sh`; + `.claude/hooks/hook-telemetry-sink.sh plugins/claude-ops/hooks/hook-telemetry-sink.sh`. +4. Fixture registry `scripts/fixtures/registry/cluster.txt` gains a commented cluster-line example; + `test_registry_filter.py` gains: cluster line excludes a canonical-plus-copies group; a member + glob matches; two instances in one carrying directory keep the group; a single-token line still + excludes exactly as before; `excluded[].path` carries the line text. +5. `reference/config.md` `duplication.registries` row and the SKILL.md configuration paragraph + describe both line shapes. + +**Files Affected** + +| File | Action | What changes | +|---|---|---| +| `plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py` | Modify | cluster grammar | +| `plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py` | Modify | five cases | +| `plugins/code-metrics/scripts/fixtures/registry/cluster.txt` | Modify | example line | +| `scripts/check-cross-plugin-source-drift.sh` | Modify | skip multi-token lines | +| `scripts/check-cross-plugin-source-drift.test.sh` | Modify | one case | +| `scripts/cross-plugin-source-registry.txt` | Modify | six cluster lines | +| `plugins/code-metrics/reference/config.md` | Modify | registries row text | + +**Sanity Check:** + +- `python3 -m unittest plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py` exits 0. +- `bash scripts/check-cross-plugin-source-drift.sh --check` exits 0 on this tree; + `bash scripts/check-cross-plugin-source-drift.test.sh` exits 0. +- Runtime probe with jscpd 5.2.0 on PATH: + `audit-duplication.sh --json --registry scripts/cross-plugin-source-registry.txt --all | jq '[.measures[]|select(any(.instances[]; .file|test("^(lib|\\.claude)/")))]|length'` + prints `0`, and `jq '[.excluded[]|select(.path|startswith("lib/hook-utils.sh"))]|length'` + prints `1` with that entry's `instances` length `17`. + +### Phase 4: Report sort, rollups, additive summary fields, summary line, no-detector headline [TODO] + +Review: code-design + +1. `report.py summarize`: when clone-group rows exist, add `by_lane` (lane to `{groups, + duplicated_lines}`) and `by_directory` (every ancestor directory of each group's first + instance, cumulative, same shape; keys are repo-relative directory paths, root as `.`). + `resummarize` inherits it, so the maps are computed over surviving groups after exclusion. +2. `report.py render`: for a document with clone-group rows, sort measures by `values.lines` + descending then tokens then first instance path (other skills keep today's sort); add a + `## Rollup` section after Measures with a per-lane table and a per-directory table listing + directories whose depth is at most `duplication.rollup_depth` (read from `thresholds`/config + passed as a new `--rollup-depth` argument, default 2); render the summary line as + `Files with clones: N.` when the document is duplication-shaped (any `instances[]` row or + `skill == "audit-duplication"`), otherwise today's line byte for byte; when every + `duplication` run row is `unavailable`, emit one headline under the title naming the first + adapter's install hint once and `/code-metrics:setup`, and render each lane row's reason with + the parenthesised hint removed. +3. `reference/report-schema.md`: document `by_lane` and `by_directory` under `summary`; add the + sentence that readers ignore unknown keys; note "intentional clones" beside "sanctioned + replication" in the `excluded` row. +4. `test_report.py`: rollup sums equal `duplicated_lines`; cumulative ancestors; depth cut in + markdown only; sort order; summary line per skill (sibling line unchanged); no-detector headline + once. + +**Files Affected** + +| File | Action | What changes | +|---|---|---| +| `plugins/code-metrics/scripts/report.py` | Modify | rollups, sort, summary line, headline | +| `plugins/code-metrics/scripts/test_report.py` | Modify | six cases | +| `plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh` | Modify | pass `--rollup-depth` | +| `plugins/code-metrics/reference/report-schema.md` | Modify | fields, ignore-unknown rule, term | + +**Sanity Check:** + +- `python3 -m unittest plugins/code-metrics/scripts/test_report.py` exits 0. +- `grep -c 'Functions:' <(audit-duplication.sh --all)` prints `0` and + `grep -c '^Files with clones:' <(audit-duplication.sh --all)` prints `1`; the sibling + `audit-size.sh --all | tail -3` still contains `Functions:`. +- `audit-duplication.sh --json --all | jq '[.summary.by_lane[].duplicated_lines]|add == .summary.duplicated_lines'` prints `true`. +- With an empty PATH prefix hiding jscpd, `audit-duplication.sh --all | grep -c 'npm install -g jscpd'` prints `1`. + +### Phase 5: SKILL.md, README, CHANGELOG, version, dogfood [TODO] + +1. `skills/audit-duplication/SKILL.md`: configuration section names the three new keys and both + registry line shapes; "Run it" gains the no-detector instruction (offer the install command to + the user, run it only on confirmation, never silently); "Reading the numbers" states clone + classes, the `partial` row, and the rollup; `## Next` kept before `## Gotchas`; the gotcha + about pairs is rewritten to state clusters. Prose stays in house style (`/ai-slop:audit` on the + file). +2. `README.md`: the audit-duplication row mentions rollups and the cluster line; the known-gaps + bullet's list of prose-restated defaults is checked against what SKILL.md now restates. +3. `CHANGELOG.md`: `## [0.1.9]` with Added / Changed / Fixed entries, one per Brief item; + `.claude-plugin/plugin.json` version 0.1.9. +4. Dogfood: `scripts/affected-tests.sh --run` over the whole diff; `scripts/run-ruff.sh check` + over the changed Python; `shellcheck` and `shfmt -d` over changed shell; both jscpd majors' + whole-tree runs recorded as distilled numbers in this file's Phase 5 notes (surviving groups, + duplicated lines, exclusions) with no memory-slice paths. + +**Files Affected** + +| File | Action | What changes | +|---|---|---| +| `plugins/code-metrics/skills/audit-duplication/SKILL.md` | Modify | config, run-it, reading, gotchas | +| `plugins/code-metrics/README.md` | Modify | row, known gaps | +| `plugins/code-metrics/CHANGELOG.md` | Modify | `[0.1.9]` | +| `plugins/code-metrics/.claude-plugin/plugin.json` | Modify | version | + +**Sanity Check:** + +- `jq -r .version plugins/code-metrics/.claude-plugin/plugin.json` prints `0.1.9`; + `bash scripts/check-changelog-parity.sh --check-bump origin/main` exits 0. +- `scripts/affected-tests.sh --run` exits 0; `scripts/run-ruff.sh check plugins/code-metrics` exits 0. +- `grep -n '^## Next' plugins/code-metrics/skills/audit-duplication/SKILL.md` precedes `^## Gotchas`. +- `markdownlint-cli2` over the changed markdown exits 0. + +### Alternatives Considered + +| Alternative | Why rejected | Switch condition | +|---|---|---| +| Detect skips as files-passed minus `statistics.total.sources` | contradicted: `sources` counts token sources that reached detection, reproduced on 4.3.0 and 5.2.0 | jscpd adds a per-file skip list to its JSON report | +| Keep jscpd's per-major defaults, detect only | jscpd 4.x users silently lose every file over 1000 lines | jscpd 4.x line reaches end of life | +| Merge pairs inside the jscpd adapter | a second pair-reporting collector would need it again; the rule is about the report | no other collector ever reports pairs and the post-pass is the only consumer | +| A second registry file for cluster lines | two registries for one concept; the plugin's docs already name this file as the shape | a second reader of the registry file cannot be taught to skip cluster lines | +| `by_directory` as direct-parent rows | no documented precedent; a plugin's files would never roll up to the plugin | a consumer needs non-overlapping per-directory sums | +| Stderr prefix as the adapter-to-dispatcher channel | `dispatch.sh` reads stderr only on failure and truncates it; a note file is unambiguous and output-testable | the adapter contract grows a structured stderr protocol for other reasons | + +### Test Strategy + +Output-based tests drive every script at its command line with fixture inputs and assert on the +printed document, per the TDD principles skill's decision order (output first, state second, +communication last). jscpd is the one unmanaged out-of-process dependency and stays stubbed by a +fake binary that replays a committed capture; nothing in-process is mocked: `cluster-clones.py`, +`registry-filter.py`, and `report.py` are driven for real by `audit-duplication.test.sh` and +`dispatch.test.sh`. One communication-based assertion exists because the boundary is the tool's +argv: `test_jscpd.py`'s existing `argv_log` stub asserts the explicit caps. Red first: each phase +writes its failing test against the named boundary, then the change. + +Test boundaries (all existing unless marked): `jscpd.py collect` argv and note-file output +(existing CLI, new env vars); `dispatch.sh` run rows (existing); `cluster-clones.py` stdin/stdout +(new script, same document contract as `registry-filter.py`); `registry-filter.py --registry` +(existing); `report.py summarize|resummarize|render` (existing, new `--rollup-depth` argument); +`audit-duplication.sh --json` (existing); `check-cross-plugin-source-drift.sh --check` (existing). +A boundary implementation picks that this list does not name is a deviation logged to +`DEVIATIONS.md` beside this file. + +Edge cases named in the Brief's criteria: overlapping-but-not-identical ranges; two copies in one +directory; all files skipped; no registry configured; a lane whose collector never resolved. +Existing tests updated: `audit-duplication.test.sh` (export assertions, three-copy case), +`test_jscpd.py` (argv), `check-cross-plugin-source-drift.test.sh` (skip case), +`test_setup_apply.py` passes unchanged once the template carries the keys. + +### Risks and Mitigations + +| Risk | Likelihood | Impact | Mitigation | +|---|---|---|---| +| jscpd 4.x argv differs from 5.x for a passed cap | Low | Med | both majors verified this session (`--max-lines`, `--max-size` share names and short forms); Phase 1's runtime probe runs both | +| A multi-token registry line breaks a reader not found in pre-flight | Low | High | pre-flight grepped all 12 mentions; the drift-checker test case is the guard; `scripts/affected-tests.sh` selects every suite referencing the registry | +| `by_directory` on a large monorepo makes the JSON heavy | Low | Low | the map holds only directories that contain a group; markdown cuts at depth 2 | +| Union-find merges two genuinely different clones that share one instance | Low | Med | the key requires identical range and equal `lines`, so only the same fragment joins | +| The sort change alters other skills' markdown | Low | Med | sort branch is gated on clone rows; `test_report.py` asserts a size document renders byte-identically | +| Version bump without changelog entry fails CI | Low | Low | Phase 5 sanity check runs the parity gate | diff --git a/docs/topics/code-metrics-duplication-audit/design/design-resolution.md b/docs/topics/code-metrics-duplication-audit/design/design-resolution.md new file mode 100644 index 0000000000..90c0fa22ef --- /dev/null +++ b/docs/topics/code-metrics-duplication-audit/design/design-resolution.md @@ -0,0 +1,86 @@ +# Design resolution: code-metrics audit-duplication fixes + +outcome: early-exit +tier: B (light design: localized contract additions inside one plugin, no new module, no topology change) +resolved: 2026-09-11, by the interview (two rounds) and five verified research runs + +## Why early-exit + +Every contract this change adds is an additive extension of a shape the plugin already has, and +the interview locked each one with its rationale and sources. No thread is open that a design +session would resolve differently from the Brief. The type sketch below is what an implementer +needs; `/planning:plan` consumes it. + +## Type sketch + +### Registry line grammar (extends `registry-filter.py`) + +```text +line := comment | blank | single | cluster +single := path-within-plugin # unchanged meaning +cluster := canonical-path SP+ member (SP+ member)* # two or more whitespace-separated tokens +member := repo-relative path | glob (pathglob.py syntax) +``` + +A group is excluded by a `cluster` line when every instance's repo-relative path matches the +canonical path or one member, and the instances sit in distinct carrying directories (the prefix in +front of the matched token; the canonical path's carrying directory is its own parent). The +`excluded[]` record keeps `{registry, line, path, instances}` with `path` = the line's text. + +### Clone-group row after clustering (unchanged schema, N instances) + +```json +{"file": null, "function": null, "lane": "bash", + "instances": [{"file": "...", "start_line": 1, "end_line": 3136}, "... N entries"], + "values": {"lines": 3136, "tokens": 25137}, + "collector": "jscpd", "labels": ["token-based", "clustered"]} +``` + +Merge key: two pair rows join when they share an instance with identical `(file, start_line, +end_line)` and equal `values.lines`. Union-find over all pair rows; `tokens` taken from the first +pair. Rows from `dupl` and `cpd` pass through untouched (already N-ary). + +### Run row for a cap-skipped lane + +```json +{"lane": "bash", "measure": "duplication", "collector": "jscpd 4.3.0", "status": "partial", + "reason": "3 of 412 files skipped by duplication.max_size 1mb / max_lines none; largest: lib/x.sh (2.1mb)"} +``` + +Adapter to dispatcher channel: the adapter writes the skip note to the path in +`CODE_METRICS_RUN_NOTE_FILE` (set by `dispatch.sh` per lane/measure/tool); when the file is +non-empty after a successful collect, `dispatch.sh` writes the run row as `partial` with that text. +When the pre-filter leaves zero files, the adapter writes the note and exits 0 without invoking the +tool; the row is `partial` and no `exit 3` occurs. + +### Summary additions (additive `code-metrics/v1`) + +```json +"summary": {"files": 432, "functions": 0, "over_reference": {}, + "duplicated_lines": 16498, "clone_groups": 705, + "by_lane": {"bash": {"groups": 380, "duplicated_lines": 12450}, "...": {}}, + "by_directory": {"plugins/code-metrics": {"groups": 127, "duplicated_lines": 1867}, "...": {}}} +``` + +`by_directory` keys are every ancestor directory of a group's first instance up to the root +(cumulative), rendered in markdown to `duplication.rollup_depth` (default 2). Both maps are +computed from surviving groups, after registry exclusion. Readers ignore unknown keys (stated in +`reference/report-schema.md`). + +### Configuration keys (`config-defaults.json`, `reference/config.md`, setup template) + +```yaml +duplication: + min_tokens: 50 + min_lines: 5 + ignore: [] + registries: [] + max_lines: null # no line cap; a number is a plugin-local guard + max_size: 1mb # jscpd 5.0.7 parser guard; SonarJS 1000kb generated-code rule + rollup_depth: 2 +``` + +Exported to adapters as `CODE_METRICS_DUP_MAX_LINES` (empty when null) and +`CODE_METRICS_DUP_MAX_SIZE`; the jscpd adapter passes both explicitly on every major, translating a +null line cap to a large explicit value on 4.x (whose `0` means "use the 1000 default") and never +emitting `--max-size 0`. From cf2bf5882a618e283260c706f15bdb7144cf9837 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 03:31:19 +0000 Subject: [PATCH 04/15] docs(code-metrics): revise the audit-duplication plan after review and stress-test Fold in the verified findings from the fresh-context plan review and the devil's-advocate pass: eighteen hook-utils instances, the 1.45mb minified bundle that the 1mb cap skips by design, the dropped telemetry-sink line (the pair differs and has no sync script), a `->` marker for cluster lines so a registered path with a space keeps working, dirname-based carrying directories, root-relative instance paths, a `partial` zero floor, an additive run-row `hint`, path-sorted merged instances, fixtures outside the scoped tree, and the affected-tests exit-3 contract. Adds blast radius, stress-test summary, execution shape, and handoff sections. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01QShQwS5yDYYToBmofdEY5N --- .../code-metrics-duplication-audit/PLAN.md | 602 +++++++++++------- .../design/design-resolution.md | 27 +- 2 files changed, 404 insertions(+), 225 deletions(-) diff --git a/docs/topics/code-metrics-duplication-audit/PLAN.md b/docs/topics/code-metrics-duplication-audit/PLAN.md index d58d6b40f9..8fdbc0a938 100644 --- a/docs/topics/code-metrics-duplication-audit/PLAN.md +++ b/docs/topics/code-metrics-duplication-audit/PLAN.md @@ -4,15 +4,26 @@ Closes melodic-software/claude-code-plugins#4068. ## Brief +Scope-change note (2026-09-11, before approval, from the fresh-context plan review and the +stress-test, each finding verified against the tree): four criteria below were corrected. The +`hook-utils.sh` class has eighteen instances (root plus seventeen plugins), not seventeen. The +shipped 1mb cap skips one tracked file here, `plugins/miro/dist/index.min.js` (1.45mb), so the +typescript lane reads `partial` by design. The `hook-telemetry-sink.sh` pair differs at line 51 +and has no sync script, so it is not sanctioned replication and its registry line is dropped +(five lines, not six). Pair-to-class merging joins byte-aligned copies; a copy embedded at a +different offset with different surrounding lines stays its own group. Cluster lines carry an +explicit `->` marker because a registered path may contain a space. + ### TLDR - `audit-duplication` reports skipped files instead of hiding them: the jscpd adapter pre-filters by byte size and line count, passes both caps explicitly on 4.x and 5.x, and marks the lane `partial` when anything was skipped; defaults are no line cap and a 1mb byte cap. -- jscpd's pair reports are merged into clone classes before summarizing, so N copies count once. -- The sanctioned-replication registry gains multi-token lines (` ...`) - so a canonical file outside any plugin can declare its copies; this repo's drift checker skips - those lines and six such lines are added here. +- jscpd's pair reports are merged into clone classes before summarizing, so byte-aligned copies + count once. +- The sanctioned-replication registry gains cluster lines + (` -> ...`) so a canonical file outside any plugin can declare + its copies; this repo's drift checker skips marked lines and five such lines are added here. - The markdown report sorts clone rows by duplicated lines, adds per-lane and per-directory rollups (cumulative, counts beside share, computed after exclusion, listed to depth 2), and the JSON gains `summary.by_lane` / `summary.by_directory` as additive `v1` fields with an explicit ignore-unknown @@ -23,11 +34,11 @@ Closes melodic-software/claude-code-plugins#4068. ### Goal A whole-tree or change-scoped duplication audit on any repository, this one included, produces -numbers a reader can act on without re-aggregating: no file is silently dropped by a size cap, a -fragment copied N times is one group counted once, replication the repository declares about itself -(including copies of a root-level canonical file) is excluded and shown as an exclusion, and the -report says where the surviving duplication sits by lane and by directory. On this repository the -audit reads clean apart from genuine duplication. +numbers a reader can act on without re-aggregating: no file is silently dropped by a size cap, +byte-aligned copies of a fragment form one group counted once, replication the repository declares +about itself (including copies of a root-level canonical file) is excluded and shown as an +exclusion, and the report says where the surviving duplication sits by lane and by directory. On +this repository the audit reads clean apart from genuine duplication. ### Constraints @@ -36,66 +47,77 @@ audit reads clean apart from genuine duplication. - The report emits no finding, severity, or exit-code gate; duplication has no reference value. - The `code-metrics/v1` schema string is unchanged; every JSON change is additive (new optional fields), and `reference/report-schema.md` states that readers ignore unknown keys. -- Single-token registry lines keep their exact meaning; the drift checker - (`scripts/check-cross-plugin-source-drift.sh`) keeps its behavior for them and skips multi-token - lines. +- Single-token registry lines keep their exact meaning, including a path that contains a space; + the drift checker (`scripts/check-cross-plugin-source-drift.sh`) keeps its behavior for them and + skips only lines carrying the `->` marker. - Both jscpd 4.x (`latest-4` = 4.3.0) and 5.x (5.2.0) stay supported by the adapter; the adapter - always passes explicit `--max-lines` and `--max-size` on 4.x (its `0` means "default" for lines and - "skip all" for size) and never emits `--max-size 0`. + always passes explicit `--max-lines` and `--max-size` on both majors, treats a configured `0` as + `null` (jscpd reads `0` as "default" for 4.x lines and "skip all" everywhere else), and never + emits a `0` cap. - An unmeasured value is `null`, never `0`; a run that measured nothing keeps "Measured nothing". - Validate with `scripts/affected-tests.sh --run`; every changed file maps to at least one suite. -- No `lib/hook-utils.sh` or other cross-plugin synced source is edited; registry lines mirror the - `src=` each `scripts/sync-*.sh` already declares. +- No `lib/hook-utils.sh` or other cross-plugin synced source is edited; every cluster line mirrors + the `src=` and copy list its `scripts/sync-*.sh` already declares. - One issue, one draft PR whose body opens with `Closes #` and carries the four required sections; CHANGELOG entry under `[0.1.9]`. ### Acceptance criteria -- Running `audit-duplication.sh --all` on this repository with jscpd 4.3.0 and again with 5.2.0 - reports identical clone groups for files under 1mb, and the run table's bash row reads `partial` - naming the count and largest of any files the adapter's pre-filter skipped; with the shipped - defaults no file in this repository is skipped. +- Running `audit-duplication.sh --json --all` on this repository with jscpd 4.3.0 and again with + 5.2.0 yields, for every byte-identical whole-file class, the same set of instances (file and + line range) per group; `tokens` and instance order are excluded from the comparison because the + two majors tokenize differently and hub on different copies. The bash row reads `ok` under both; + the typescript row reads `partial` under both, naming `plugins/miro/dist/index.min.js` as the one + file over the 1mb cap. - `duplication.max_lines` defaults to `null` (no cap) and `duplication.max_size` to `1mb`; both are documented in `reference/config.md` (gated against `config-defaults.json`) and exported to the - adapter, and the adapter's tests cover the 4.x explicit-cap translation and the never-`0` rule. + adapter, and the adapter's tests cover the explicit-cap argv on both majors, the `0`-means-null + rule, and that no `0` cap is ever passed. - IF every file in a lane is skipped by the caps, THEN that lane's run row reads `partial` with the reason, no collector is invoked for it, and the script never exits 3 for that cause. -- Seventeen identical copies of `hook-utils.sh` (root `lib/` plus sixteen plugins) produce exactly - one clone group with seventeen instances, and with the registry line - `lib/hook-utils.sh plugins/*/hooks/hook-utils.sh` that group appears once under `excluded[]` - with `duplicated_lines` counted once, not sixteen times. +- The eighteen byte-identical copies of `hook-utils.sh` (root `lib/` plus the seventeen plugin + copies `scripts/sync-hook-utils.sh --print-manifest` lists) produce exactly one clone group with + eighteen instances, and with the registry line + `lib/hook-utils.sh -> plugins/*/hooks/hook-utils.sh` that group appears once under `excluded[]` + with `duplicated_lines` counted once. - Two groups whose instances overlap without identical line ranges (the `hook-telemetry-sink.sh` - shape) stay separate groups after the merge. -- A multi-token registry line excludes a group only when every instance matches the canonical path - or one of the copy paths or globs and the instances sit in distinct carrying directories; two copies - inside one directory still count as duplication; a single-token line behaves exactly as before - (existing registry tests unchanged and passing). -- `scripts/check-cross-plugin-source-drift.sh --check` passes on this repository with the six new - multi-token lines present, and its tests cover a multi-token line being skipped. -- After this change, `audit-duplication.sh --registry scripts/cross-plugin-source-registry.txt --all` - on this repository reports zero surviving groups whose instances include a file under root `lib/` - or `.claude/hooks/`. + shape, and three copies of one fragment embedded at different offsets) stay separate groups after + the merge. +- A cluster line excludes a group only when every instance's root-relative path matches the + canonical path or one of the members (literal or glob) and the instances' directories are all + distinct; two copies inside one directory still count as duplication; a single-token line behaves + exactly as before, a registered path containing a space included; when a single-token line and a + cluster line both match, the first matching line in file order wins. +- `scripts/check-cross-plugin-source-drift.sh --check` passes on this repository with the five new + cluster lines present, each under its own annotation block, and its tests cover a marked line + being skipped. +- After this change, `audit-duplication.sh --json --registry scripts/cross-plugin-source-registry.txt --all` + on this repository reports zero surviving groups whose instances include a file under root `lib/`, + from the repository root and from a subdirectory alike. - WHILE no registry is configured and none is passed, the report's `excluded[]` is empty and the - summary line says so. + summary states that no registry was configured. - The markdown Measures table for a duplication document lists clone groups in descending order of duplicated lines, and the report carries a `## Rollup` section with a per-lane table and a per-directory table (rows to depth 2 by default, `duplication.rollup_depth` configurable) whose - numbers are cumulative up the tree, carry `groups` and `duplicated_lines`, and sum consistently - with `summary.duplicated_lines` after exclusion. -- The JSON `summary` carries `by_lane` and `by_directory` with the same numbers; `schema` is still - `code-metrics/v1`; `reference/report-schema.md` documents both fields and states that readers - ignore unknown keys; `verification:measure`'s consumption is unaffected. + numbers are cumulative up the tree and carry `groups` and `duplicated_lines`; the per-lane + values sum to `summary.duplicated_lines`, and the root row of `by_directory` equals it. +- The JSON `summary` carries `by_lane` and `by_directory` with the same numbers (empty maps when a + duplication collector ran and found nothing); `schema` is still `code-metrics/v1`; + `reference/report-schema.md` documents both fields, the run row's additive `hint` field, and + states that readers ignore unknown keys; `verification:measure`, the one marketplace consumer, + reads only `status` and is unaffected. - The duplication summary line reads `Files with clones: N.` followed by the duplicated-lines and exclusion lines, with no "Functions" or "Over reference" text; the sibling skills' summary lines are byte-identical to today's. - When no duplication collector resolves, the markdown opens with one headline naming the install - command and `/code-metrics:setup`, the per-lane rows no longer repeat the install hint, and - SKILL.md instructs Claude to offer the install and run it only on confirmation. + command and `/code-metrics:setup`, taken from the run row's `hint` field rather than parsed out of + its reason, and SKILL.md instructs Claude to offer the install and run it only on confirmation. - The jscpd adapter's docstring and `reference/collectors.md` state that 4.x and 5.x are both - translated, pin 5.2.0, and note the `kind` field; the schema reference names "intentional clones" - beside "sanctioned replication". + translated, pin 5.2.0, note the `kind` field, and record that the two majors tokenize differently; + the schema reference names "intentional clones" beside "sanctioned replication". - `plugin.json` reads 0.1.9 and `CHANGELOG.md` carries a `[0.1.9]` entry covering every item above; - `scripts/affected-tests.sh --run` passes. + `scripts/affected-tests.sh --run` exits 0, or exits 3 with only Python suites listed as not run, + each of which then passes under pytest. ### Captured assumptions @@ -105,17 +127,28 @@ audit reads clean apart from genuine duplication. - Once-per-class counting follows PMD CPD and SonarQube; the fetched literature is silent on how duplicated lines are totalled, and jscpd v5 is the one tool that sums per pair. Revisit if a standard sets a duplicated-lines definition. -- The merge keys on identical (file, start_line, end_line) instances and equal `lines`; closure is - exact only for type-1/type-2 clones, which is all the adapter receives today because it passes no - `--max-gap-lines`. Revisit if `similar` clones are ever enabled. +- The merge keys on identical (file, start_line, end_line) instances and equal `lines`, so only + byte-aligned copies join; jscpd extends a clone greedily into shared flanking lines, so offset + copies get ranges differing by a line and stay separate on both majors. Closure is exact only for + type-1/type-2 clones, which is all the adapter receives because it passes no `--max-gap-lines`. + Revisit if `similar` clones are ever enabled. +- Merged instances are sorted by path so the first instance, and therefore `by_directory` + attribution, is the same under 4.x (which hubs on the last input) and 5.x (which hubs on the first). - `rollup_depth` default 2 is the plugin's choice; no upstream sets a depth (SonarQube and Codacy roll up every directory). Revisit if a consuming repository's layout makes depth 2 meaningless. - The byte cap of 1mb aligns with jscpd 5.0.7's parser guard and SonarJS's 1000kb generated-code - rule; no line cap by default aligns with jscpd 5, PMD CPD, SonarQube, and Linguist. Revisit if - jscpd changes its default in a later major. + rule, and means 1,048,576 bytes, the value jscpd 5.2.0 reports for `1mb`; no line cap by default + aligns with jscpd 5, PMD CPD, SonarQube, and Linguist. Revisit if jscpd changes its default or + multiplier in a later major. - The cluster lines live in the existing registry file because the plugin's own documentation names that file as the shape; this is a repository-convention choice with no external authority. Revisit if the drift checker grows a second consumer of the file. +- The `->` marker is the cluster-line signal because a registered single-token path may contain a + space and the drift checker's tests protect that case. Revisit if a consuming repository has a + path containing ` -> `. +- The `.claude/hooks/hook-telemetry-sink.sh` and `plugins/claude-ops/hooks/hook-telemetry-sink.sh` + pair differs at line 51 and has no sync script, so the audit keeps reporting its overlap as + duplication; a future sync script earns it a cluster line. Revisit when that script exists. - "Sanctioned replication" stays the plugin's term; the literature's term "intentional clones" (Cordy 2008) is named beside it. No upstream tool models the canonical-plus-copies relation, so no vocabulary conflict exists. @@ -131,6 +164,8 @@ audit reads clean apart from genuine duplication. steps unchanged. - A second registry file or a manifest generated from the sync scripts. - Migrating the registry's single-token lines to the new grammar. +- A repository config excluding `plugins/miro/dist/` from this repo's own audits; the `partial` + row is the designed reading and a config is the consuming repo's choice. ### Deferred questions @@ -141,8 +176,8 @@ audit reads clean apart from genuine duplication. ### Goal **What**: the six audit-duplication fixes the Brief locks, in `plugins/code-metrics` at 0.1.9, -plus one drift-checker change and six registry lines in this repository, in one draft PR that -closes issue 4068. +plus one drift-checker change and five registry cluster lines in this repository, in one draft PR +that closes issue 4068. **Why**: the whole-tree audit hid skipped files behind `complete`, inflated every count by pair reporting, could not declare this repo's own canonical copies, and left the reader to aggregate @@ -151,8 +186,8 @@ by hand; each number the skill reports has to be one a reader can act on. ### Standards grounding No `.claude/standards.yaml` and no `docs/standards/README.md` exist, so the ladder's rung 4 -applied: inferred from repository conventions not auto-loaded. Offer stands to bootstrap an index -through the planning setup; nothing was written. +applied: inferred from repository conventions not auto-loaded. The offer to bootstrap an index +through the planning setup stands; nothing was written. | Surface | Sections cited | Layer provenance | |---|---|---| @@ -160,98 +195,132 @@ through the planning setup; nothing was written. | Skill body | `.claude/rules/skill-bodies-state-current-rules.md` (current rule and reason, no incident narration, `## Next` before `## Gotchas`); `.claude/rules/vendor-docs-are-not-style.md` (house style, `/ai-slop:audit`) | team, path-scoped | | Shell tests | `docs/conventions/shell-test-helpers/README.md` (per-plugin `pass`/`fail` helpers stay per plugin; repo-tooling suites use `scripts/lib/test-harness.sh`) | team | | Upstream facts | `docs/conventions/upstream-drift/README.md` (a restated upstream specific carries claim, basis, as-of date, recheck trigger) | team | -| Validation | `AGENTS.md` "Validate a change" (`scripts/affected-tests.sh --run`; a changed file mapping to zero suites is an error) | team, ambient | +| Validation | `AGENTS.md` "Validate a change" (`scripts/affected-tests.sh --run`; exit 3 lists suites in ecosystems it cannot run) | team, ambient | | Release | `scripts/check-changelog-parity.sh --check-bump` (a manifest bump must add a `## []` entry) | team | ### Approach Five phases, integration slice first. Phase 1 lands the cap machinery end to end (config key to -run row) because it is the only phase that changes what jscpd is asked to scan, and everything -after it measures the same scope. Phase 2 and Phase 3 are file-disjoint and together satisfy the -Brief's headline criterion (one 17-instance `hook-utils.sh` group, excluded once). Phase 4 is the -rendering layer over the shape Phases 2 and 3 produce. Phase 5 is docs and release. - -Build technique: kept tracer-bullet slice (Phase 1 runs `audit-duplication.sh --all` on this -repository under both jscpd majors as its runtime probe); no throwaway spike, since the research +run row) because it is the only phase that changes what jscpd is asked to scan. Phases 2 and 3 +together satisfy the headline criterion (one eighteen-instance `hook-utils.sh` group, excluded +once). Phase 4 is the rendering layer over the shape Phases 2 and 3 produce. Phase 5 is docs and +release. Phase 3 is committed on its own because editing the registry fans the CI test selection +out to most of the corpus, and a red there should bisect to one commit. + +Build technique: kept tracer-bullet slice (Phase 1's runtime probe runs the audit on this +repository under both jscpd majors); no throwaway spike, since the research and the stress-test already reproduced every tool behavior the design relies on. +Pre-flight results (done during planning, recorded so no phase repeats them): the registry file is +named in twelve files; only `scripts/check-cross-plugin-source-drift.sh` parses it into a lookup, +`scripts/check-shell-portability.sh` compares whole lines to a path-within-plugin and cannot match +a marked line, every other mention is a comment. No script outside `plugins/code-metrics/` reads +`code-metrics/v1` documents; `verification:measure` reads `status` only. + +Tool provisioning for probes: jscpd is never installed by the plugin, but the runtime probes and +the fixture captures need both majors. A session runs +`npm install --prefix /jscpd4 jscpd@4.3.0` and `npm install --prefix /jscpd5 jscpd@5.2.0` +into two scratch prefixes outside the repository and prepends the wanted `node_modules/.bin` to +`PATH` per probe. A probe whose major is absent prints `SKIP` and is not a failure, the way +`audit-duplication.test.sh`'s real-cluster case already does. + ### Phase 1: Explicit caps, adapter pre-filter, partial run row [TODO] Review: code-design 1. Add `duplication.max_lines` (`null`), `duplication.max_size` (`"1mb"`), and `duplication.rollup_depth` (`2`) to `scripts/config-defaults.json`; add the three rows to - `reference/config.md` (the gate `scripts/check-code-metrics-config-reference.py` pins key set - and default rendering); add the three keys to `skills/setup/templates/config-template.yaml` - (pinned leaf for leaf by `test_setup_apply.py`). -2. `audit-duplication.sh`: read the two caps beside the existing tunables and export + `reference/config.md` (gated by `scripts/check-code-metrics-config-reference.py`), with the + note that a `0` cap means `null` and that CRLF checkouts count one extra byte per line; add the + three keys to `skills/setup/templates/config-template.yaml` (pinned by `test_setup_apply.py`). +2. `audit-duplication.sh`: read the caps beside the existing tunables, map `0` to null, and export `CODE_METRICS_DUP_MAX_LINES` (empty when null) and `CODE_METRICS_DUP_MAX_SIZE`; keep the existing three exports byte-identical. -3. `dispatch.sh`: before each `collect`, export `CODE_METRICS_RUN_NOTE_FILE` pointing at - `$WORK/note.$lane.$measure.$tool`; after a `collect` that exits 0, if that file is non-empty, - write the run row as `partial` with the file's single line as the reason, else `ok` as today. +3. `dispatch.sh`: before each `collect`, export `CODE_METRICS_PARTIAL_REASON_FILE` pointing at + `$WORK/partial.$lane.$measure.$tool` (the work dir is a fresh `mktemp -d` per run and one tool + runs per lane, so no stale file exists); after a `collect` that exits 0, if that file is + non-empty, write the run row as `partial` with the file's single line as the reason, else `ok` + as today. On a failed probe, also write the adapter's install hint into an additive `hint` + field on the run row (`null` otherwise) and keep `reason` as it is built today. [EXEC-SHAPE] File-per-channel matches how `dispatch.sh` already isolates each adapter's stdout and stderr into `$WORK` files. -4. `collectors/jscpd.py`: parse `CODE_METRICS_DUP_MAX_SIZE` with jscpd's own unit grammar - (`kb`/`mb`/raw bytes) and `CODE_METRICS_DUP_MAX_LINES`; pre-filter the file list by `os.stat` - size and newline count; pass `--max-size ` always and `--max-lines ` - always (4.x reads `0` as "use default", so the null cap is spelled as a large explicit number - [EXEC-SHAPE]); never emit `--max-size 0`; when files were skipped, write one line to the note - file (`N of M files skipped by duplication.max_size / max_lines ; largest: - ()`); when the pre-filter leaves zero files, write the note and return 0 without invoking +4. `collectors/jscpd.py`: parse `CODE_METRICS_DUP_MAX_SIZE` with the multipliers jscpd uses + (`kb` = 1024, `mb` = 1,048,576, bare digits = bytes) and `CODE_METRICS_DUP_MAX_LINES`; treat an + empty or `0` value as no cap; pre-filter the file list by `os.stat` size and, only when a line + cap is set, by a binary-mode newline count; pass `--max-size ` always and + `--max-lines ` always, so the pre-filter is the only gate on both + majors [EXEC-SHAPE] (4.x reads `--max-lines 0` as "use the 1000 default" and both majors read a + `0` size as "skip all"); when files were skipped, write one line to the partial-reason file + (`N of M files skipped by duplication.max_size / max_lines ; largest: ()`), + or to stderr when the variable is unset (direct runs) or the path is unwritable, never failing + for that; when the pre-filter leaves zero files, write the note and return 0 without invoking jscpd (4.x would write no report and the current exit-3 path would call that a failure). Extend - the `probe` docstring and module docstring: 4.x and 5.x both translate; drop the "jscpd 4 is not - translated" sentence; record the 5.2.0 `kind` field. -5. `reference/collectors.md`: jscpd row pinned to 5.2.0 with a fresh as-of date and the recheck - trigger unchanged; add the 4.x maintenance-line fact (`latest-4` = 4.3.0) as a four-part - verification record. + the module docstring: 4.x and 5.x both translate; drop the "jscpd 4 is not translated" + sentence; record the 5.2.0 `kind` field and that the majors tokenize differently. +5. `registry-filter.py --zero-floor`: a `partial` duplication row counts as measured, so an + all-excluded or clone-free lane that skipped a file still states `duplicated_lines: 0` and + `clone_groups: 0`. +6. `reference/collectors.md`: jscpd row pinned to 5.2.0 with a fresh as-of date; add four-part + verification records for the 4.x maintenance line (`latest-4` = 4.3.0), the `1mb` multiplier, + and the token-count difference between majors. **Files Affected** | File | Action | What changes | |---|---|---| | `plugins/code-metrics/scripts/config-defaults.json` | Modify | three keys under `duplication` | -| `plugins/code-metrics/reference/config.md` | Modify | three key rows | +| `plugins/code-metrics/reference/config.md` | Modify | three key rows, `0`-means-null and CRLF notes | | `plugins/code-metrics/skills/setup/templates/config-template.yaml` | Modify | three keys | -| `plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh` | Modify | two exports | -| `plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh` | Modify | export assertions | -| `plugins/code-metrics/scripts/dispatch.sh` | Modify | note-file channel, `partial` row | -| `plugins/code-metrics/scripts/dispatch.test.sh` | Modify | partial-row case via the real jscpd adapter and a stub binary | -| `plugins/code-metrics/scripts/collectors/jscpd.py` | Modify | caps, pre-filter, note file, docstring | -| `plugins/code-metrics/scripts/collectors/test_jscpd.py` | Modify | argv, skip, all-skipped, size-grammar cases | -| `plugins/code-metrics/reference/collectors.md` | Modify | jscpd row, 4.x record | +| `plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh` | Modify | two exports, `0` to null | +| `plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh` | Modify | export assertions; its `5.1.2` stub version string becomes `5.2.0` | +| `plugins/code-metrics/scripts/dispatch.sh` | Modify | partial-reason channel, `partial` row, `hint` field | +| `plugins/code-metrics/scripts/dispatch.test.sh` | Modify | partial-row case: sets `CODE_METRICS_DUP_MAX_SIZE` to a small value itself (dispatch never exports caps) and adds a jscpd stub beside the existing scc stub; a `hint` assertion on a failed probe | +| `plugins/code-metrics/scripts/collectors/jscpd.py` | Modify | caps, pre-filter, note, docstring | +| `plugins/code-metrics/scripts/collectors/test_jscpd.py` | Modify | argv on both stub versions, skip, all-skipped, size grammar, `0`, unset variable; docstring version | +| `plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py` | Modify | zero floor counts `partial` | +| `plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py` | Modify | zero-floor `partial` case | +| `plugins/code-metrics/reference/collectors.md` | Modify | jscpd row, three verification records | **Sanity Check:** -- `python3 scripts/check-code-metrics-config-reference.py` exits 0. -- `python3 -m unittest plugins/code-metrics/scripts/collectors/test_jscpd.py` exits 0 and its - argv-log case asserts `--max-size 1mb` and `--max-lines 1000000` present and `--max-size 0` absent. -- `bash plugins/code-metrics/scripts/dispatch.test.sh` exits 0 with a case whose JSON has - `run[].status == "partial"` and a reason matching `^[0-9]+ of [0-9]+ files skipped`. -- Runtime probe: with jscpd 5.2.0 on PATH, - `audit-duplication.sh --json --registry scripts/cross-plugin-source-registry.txt --all | jq '.run[]|select(.lane=="bash")|.status'` - prints `"ok"` (no file in this repo exceeds 1mb), and the same command with jscpd 4.3.0 on PATH - prints `"ok"` with `summary.files` equal between the two runs. +- `python3 scripts/check-code-metrics-config-reference.py` exits 0; + `python3 -m unittest plugins/code-metrics/skills/setup/scripts/test_setup_apply.py` exits 0. +- `python3 -m unittest plugins/code-metrics/scripts/collectors/test_jscpd.py` exits 0; its argv-log + case asserts `--max-size 1048577` and `--max-lines 1000000` present and no argument equal to `0` + follows either flag. +- `bash plugins/code-metrics/scripts/dispatch.test.sh` exits 0 with a case whose JSON has a run row + `status == "partial"` and a reason matching `^[0-9]+ of [0-9]+ files skipped`, and a case whose + failed-probe row carries a non-null `hint`. +- `python3 -m unittest plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py` + exits 0 with the zero-floor `partial` case. +- Runtime probe (SKIP when a major is absent): with jscpd 5.2.0 on PATH, + `audit-duplication.sh --json --all | jq -c '[.run[]|select(.measure=="duplication")|{lane,status}]'` + shows `bash` `ok` and `typescript` `partial`, and the typescript reason names + `plugins/miro/dist/index.min.js`; the same under 4.3.0. ### Phase 2: Merge pairs into clone classes [TODO] Review: code-design 1. New `skills/audit-duplication/scripts/cluster-clones.py` (stdin document, stdout document): - union-find over rows carrying exactly two `instances[]` whose `collector` reports pairs (rows - with three or more instances pass through untouched, which covers `dupl` and `cpd`); two rows - join when they share an instance with identical `(file, start_line, end_line)` and equal - `values.lines`; the merged row keeps the first row's `values`, the union of instances in - first-seen order, and appends `clustered` to `labels`. Rows without `instances` pass through. - Exit 0 on a printed document, 2 on a non-JSON stdin. + union-find over every row with exactly two `instances[]`, whatever its `collector` (a + three-or-more-instance row is already a class and passes through); two rows join when they + share an instance with identical `(file, start_line, end_line)` and equal `values.lines`; the + merged row keeps the first row's `values`, the union of instances sorted by `(file, + start_line)`, and appends `clustered` to `labels`. Rows without `instances` pass through. Exit 0 + on a printed document, 2 on a non-JSON stdin. 2. `audit-duplication.sh`: pipe `report.json` through `cluster-clones.py` before `registry-filter.py`. -3. Fixture: add `scripts/fixtures/sources/cluster/gamma/shared/shared-utils.sh` (byte-identical - third copy) and a committed capture `scripts/fixtures/tool-output/jscpd-three.json` produced by - a real jscpd 5.2.0 run over the three copies (two pair rows against `alpha`); the existing - two-copy capture and every test that replays it stay unchanged. -4. `test_cluster_clones.py`: three copies collapse to one three-instance group counted once; - overlapping-but-not-identical ranges stay separate; a three-instance input row passes through; - `summary` is left to `report.py resummarize`. +3. Fixtures, outside `fixtures/sources` so no suite that scopes that tree changes its counts: + `scripts/fixtures/clone-classes/aligned/{a,b,c}/shared/shared-utils.sh` (three byte-identical + copies) and `scripts/fixtures/clone-classes/offset/{c1,c2,c3}.sh` (one 41-line fragment at + offsets 1, 2, 3 with different flanking lines); committed captures + `scripts/fixtures/tool-output/jscpd-aligned3.json` and `jscpd-offset3.json` produced by a real + jscpd 5.2.0 run, then rewritten to repo-relative names (the adapter passes `--absolute`, so the + raw capture carries machine paths, and the stub replays the file regardless of input). +4. `test_cluster_clones.py`: aligned three copies collapse to one three-instance group with + `lines` counted once; offset copies stay two groups; a three-instance input row passes + through; a two-instance `cpd`-labelled row joins when it shares an identical instance; merged + instance order is by path; `summary` is left to `report.py resummarize`. **Files Affected** @@ -260,131 +329,164 @@ Review: code-design | `plugins/code-metrics/skills/audit-duplication/scripts/cluster-clones.py` | Create | the post-pass | | `plugins/code-metrics/skills/audit-duplication/scripts/test_cluster_clones.py` | Create | output-based tests | | `plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh` | Modify | pipeline step | -| `plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh` | Modify | three-copy case through the stub | -| `plugins/code-metrics/scripts/fixtures/sources/cluster/gamma/shared/shared-utils.sh` | Create | third copy | -| `plugins/code-metrics/scripts/fixtures/tool-output/jscpd-three.json` | Create | capture | +| `plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh` | Modify | aligned and offset cases through the stub | +| `plugins/code-metrics/scripts/fixtures/clone-classes/aligned/{a,b,c}/shared/shared-utils.sh` | Create | three copies | +| `plugins/code-metrics/scripts/fixtures/clone-classes/offset/{c1,c2,c3}.sh` | Create | offset copies | +| `plugins/code-metrics/scripts/fixtures/tool-output/jscpd-aligned3.json` | Create | capture, relative names | +| `plugins/code-metrics/scripts/fixtures/tool-output/jscpd-offset3.json` | Create | capture, relative names | +| `plugins/code-metrics/scripts/dispatch.test.sh` | KEEP | its `summary.files == 7` assertion over `fixtures/sources` is untouched by the new fixture tree | **Sanity Check:** - `python3 -m unittest plugins/code-metrics/skills/audit-duplication/scripts/test_cluster_clones.py` exits 0. - `bash plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh` exits 0 with a case asserting `summary.clone_groups == 1` and `len(measures[0].instances) == 3` on the - three-copy capture with no registry. -- `cmp` the three fixture copies: identical. + aligned capture with no registry, and `summary.clone_groups == 2` on the offset capture. +- `cmp` the three aligned copies pairwise: identical; + `grep -c '^/' plugins/code-metrics/scripts/fixtures/tool-output/jscpd-aligned3.json` prints `0`. +- `bash plugins/code-metrics/scripts/dispatch.test.sh` exits 0 unchanged. -### Phase 3: Registry cluster lines, drift checker, this repo's six lines [TODO] +### Phase 3: Registry cluster lines, drift checker, this repo's five lines [TODO] Review: code-design -1. `registry-filter.py`: `read_registry` returns, per line, either a single token or a - `(canonical, [members...])` tuple; `sanctions()` for a cluster line matches each instance's - repo-relative path against the canonical path or any member via `pathglob.matches`, requires - distinct carrying directories (the prefix in front of the matched token; the canonical path's - parent for itself), and records `path` as the line's text. Single-token behavior byte-identical. - Pre-flight (done in planning): the only other parser of the registry file is - `scripts/check-cross-plugin-source-drift.sh`; `check-shell-portability.sh` compares whole lines - to a path-within-plugin and cannot match a multi-token line; every other mention is a comment. +Committed on its own (the registry edit fans CI's test selection out to roughly 225 suites). + +1. `registry-filter.py`: a line containing ` -> ` is a cluster line: the text before the arrow is + the canonical path, the whitespace-separated tokens after it are members (literal paths or + `pathglob` globs); every other non-comment line is a single token taken whole, spaces included. + Before matching, normalize every instance path to root-relative by joining the cwd-relative + value onto the cwd and taking `relpath` against `--root` (the dispatcher rebases paths onto the + cwd, and an anchored glob rejects a `../` prefix). A cluster line sanctions a group when every + normalized instance equals the canonical path or matches one member, and the instances' + `dirname`s are pairwise distinct (the glob matcher anchors the whole path, so "prefix before the + token" is empty for a glob and the single-token prefix rule cannot be reused). `excluded[].path` + carries the line text. The first matching line in file order wins, stated in the docstring and + `reference/config.md`. 2. `scripts/check-cross-plugin-source-drift.sh`: in the registry load loop, `continue` on a line - containing whitespace after trimming, with a comment naming the cluster-line grammar and its - owner (the code-metrics registry filter). `check-cross-plugin-source-drift.test.sh`: a case - where a multi-token line neither registers nor reports `REGISTRY STALE`. -3. `scripts/cross-plugin-source-registry.txt`: a commented section "Canonical sources outside a - plugin (cluster lines; read by code-metrics audit-duplication, skipped by the drift checker)" - with six lines, each mirroring its sync script's `src=` and copy paths: - `lib/hook-utils.sh plugins/*/hooks/hook-utils.sh`; - `lib/rewrite-guard.sh plugins/*/hooks/rewrite-guard.sh`; - `lib/index-regen.sh plugins/*/scripts/index-regen.sh`; - `lib/resolve-convention-pattern.sh plugins/*/hooks/resolve-convention-pattern.sh`; - `lib/parse-concern-value.sh plugins/*/skills/*/scripts/parse-concern-value.sh plugins/*/skills/*/scripts/lib/parse-concern-value.sh`; - `.claude/hooks/hook-telemetry-sink.sh plugins/claude-ops/hooks/hook-telemetry-sink.sh`. -4. Fixture registry `scripts/fixtures/registry/cluster.txt` gains a commented cluster-line example; - `test_registry_filter.py` gains: cluster line excludes a canonical-plus-copies group; a member - glob matches; two instances in one carrying directory keep the group; a single-token line still - excludes exactly as before; `excluded[].path` carries the line text. -5. `reference/config.md` `duplication.registries` row and the SKILL.md configuration paragraph - describe both line shapes. + containing ` -> `, with a comment naming the cluster-line grammar and its reader. + `check-cross-plugin-source-drift.test.sh`: a marked line neither registers nor reports + `REGISTRY STALE`; the existing space-bearing-path case stays green. +3. `scripts/cross-plugin-source-registry.txt`: header line "One path-within-plugin per line" + gains the cluster-line sentence; five cluster lines, each under its own annotation block naming + its dedicated check (the production-registry policy test resets its comment block after every + entry): `lib/hook-utils.sh -> plugins/*/hooks/hook-utils.sh` (`scripts/sync-hook-utils.sh --check`); + `lib/rewrite-guard.sh -> plugins/*/hooks/rewrite-guard.sh` (`scripts/sync-rewrite-guard.sh --check`); + `lib/index-regen.sh -> plugins/*/scripts/index-regen.sh` (`scripts/sync-index-regen.sh --check`); + `lib/resolve-convention-pattern.sh -> plugins/*/hooks/resolve-convention-pattern.sh` + (`scripts/sync-resolve-convention-pattern.sh --check`); + `lib/parse-concern-value.sh -> plugins/*/skills/*/scripts/parse-concern-value.sh plugins/*/skills/*/scripts/lib/parse-concern-value.sh` + (`scripts/sync-parse-concern-value.sh --check`). Before committing, run the Phase 3 probe below + and add a line only for a surviving root `lib/` group that a sync script declares. +4. Fixture registry `scripts/fixtures/registry/cluster.txt` gains a commented cluster-line example + and its header sentence; `test_registry_filter.py` gains: a cluster line excludes a + canonical-plus-copies group; a glob member matches; two instances in one directory keep the + group; a single-token path containing a space still matches whole; first matching line wins + when both shapes match; instances given cwd-relative from a subdirectory still match. +5. Prose that restates the grammar: `reference/config.md` `duplication.registries` row, + `plugins/claude-config/skills/audit-pass/reference/exclusion-set.md` line 18 ("entries are paths + within each plugin"), and the SKILL.md configuration paragraph (Phase 5). **Files Affected** | File | Action | What changes | |---|---|---| -| `plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py` | Modify | cluster grammar | -| `plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py` | Modify | five cases | -| `plugins/code-metrics/scripts/fixtures/registry/cluster.txt` | Modify | example line | -| `scripts/check-cross-plugin-source-drift.sh` | Modify | skip multi-token lines | +| `plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py` | Modify | cluster grammar, root normalization, precedence | +| `plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py` | Modify | six cases | +| `plugins/code-metrics/scripts/fixtures/registry/cluster.txt` | Modify | example line, header | +| `scripts/check-cross-plugin-source-drift.sh` | Modify | skip marked lines | | `scripts/check-cross-plugin-source-drift.test.sh` | Modify | one case | -| `scripts/cross-plugin-source-registry.txt` | Modify | six cluster lines | +| `scripts/cross-plugin-source-registry.txt` | Modify | header, five annotated cluster lines | | `plugins/code-metrics/reference/config.md` | Modify | registries row text | +| `plugins/claude-config/skills/audit-pass/reference/exclusion-set.md` | Modify | one sentence | **Sanity Check:** - `python3 -m unittest plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py` exits 0. - `bash scripts/check-cross-plugin-source-drift.sh --check` exits 0 on this tree; - `bash scripts/check-cross-plugin-source-drift.test.sh` exits 0. -- Runtime probe with jscpd 5.2.0 on PATH: - `audit-duplication.sh --json --registry scripts/cross-plugin-source-registry.txt --all | jq '[.measures[]|select(any(.instances[]; .file|test("^(lib|\\.claude)/")))]|length'` + `bash scripts/check-cross-plugin-source-drift.test.sh` exits 0 (including the production-registry + policy case). +- Runtime probe (SKIP when absent) with jscpd 5.2.0 on PATH, from the repository root: + `audit-duplication.sh --json --registry scripts/cross-plugin-source-registry.txt --all | jq '[.measures[]|select(any(.instances[]; .file|test("^lib/")))]|length'` prints `0`, and `jq '[.excluded[]|select(.path|startswith("lib/hook-utils.sh"))]|length'` - prints `1` with that entry's `instances` length `17`. + prints `1` with that entry's `instances` length equal to + `$(scripts/sync-hook-utils.sh --print-manifest | grep -c copy) + 1`; the same two commands run + from `plugins/code-metrics` with `--registry ../../scripts/cross-plugin-source-registry.txt` + print the same values. ### Phase 4: Report sort, rollups, additive summary fields, summary line, no-detector headline [TODO] Review: code-design -1. `report.py summarize`: when clone-group rows exist, add `by_lane` (lane to `{groups, - duplicated_lines}`) and `by_directory` (every ancestor directory of each group's first - instance, cumulative, same shape; keys are repo-relative directory paths, root as `.`). - `resummarize` inherits it, so the maps are computed over surviving groups after exclusion. -2. `report.py render`: for a document with clone-group rows, sort measures by `values.lines` - descending then tokens then first instance path (other skills keep today's sort); add a +1. `report.py summarize` gains an optional `--root`: when clone-group rows exist, add `by_lane` + (lane to `{groups, duplicated_lines}`) and `by_directory` (every ancestor directory of each + group's first instance after root-normalization, cumulative, root as `.`, same shape); emit + both as empty maps when a duplication collector ran and no group survived. `assemble` and + `resummarize` accept `--root`; `audit-duplication.sh` passes it to `resummarize`, so the maps + are computed over surviving groups after exclusion. A group is attributed to its first + instance's ancestors (instances are path-sorted by Phase 2), so the identity that holds is + `by_directory["."]["duplicated_lines"] == summary.duplicated_lines` and the per-lane sum; rows + below the root cannot be summed, which the schema reference states. +2. `report.py assemble`: a `partial` run row counts as measured for document `status`, so an + all-skipped lane yields `partial`, not `empty`, and the `Unavailable:` line is followed by a + `Partial:` line naming lanes that skipped files. +3. `report.py render`: for a document with clone-group rows, sort measures by `values.lines` + descending, then `tokens`, then first instance path (other skills keep today's sort); add a `## Rollup` section after Measures with a per-lane table and a per-directory table listing - directories whose depth is at most `duplication.rollup_depth` (read from `thresholds`/config - passed as a new `--rollup-depth` argument, default 2); render the summary line as - `Files with clones: N.` when the document is duplication-shaped (any `instances[]` row or - `skill == "audit-duplication"`), otherwise today's line byte for byte; when every - `duplication` run row is `unavailable`, emit one headline under the title naming the first - adapter's install hint once and `/code-metrics:setup`, and render each lane row's reason with - the parenthesised hint removed. -3. `reference/report-schema.md`: document `by_lane` and `by_directory` under `summary`; add the - sentence that readers ignore unknown keys; note "intentional clones" beside "sanctioned - replication" in the `excluded` row. -4. `test_report.py`: rollup sums equal `duplicated_lines`; cumulative ancestors; depth cut in - markdown only; sort order; summary line per skill (sibling line unchanged); no-detector headline - once. + directories whose depth is at most `--rollup-depth` (default 2; `audit-duplication.sh` passes + the resolved key); render the summary line as `Files with clones: N.` when the document is + duplication-shaped (any `instances[]` row or `skill == "audit-duplication"`), otherwise today's + line byte for byte; when a duplication document has an empty `excluded[]`, print + `Excluded by a sanctioned-replication registry: 0 (no registry configured).`; when every + `duplication` run row is `unavailable`, emit one headline under the title with the first + non-null `hint` and `/code-metrics:setup`, and print each lane row's reason unchanged. +4. `reference/report-schema.md`: document `by_lane`, `by_directory` (attribution rule, root + identity, empty-map floor), the run row's `hint`, the `partial` document status for a lane that + skipped files; add the sentence that readers ignore unknown keys; note "intentional clones" + beside "sanctioned replication" in the `excluded` row. +5. `test_report.py`: rollup sums and root identity; cumulative ancestors; depth cut in markdown + only; empty maps on a clone-free duplication run; sort order; summary line per skill (the + existing exact-dict assertion for a size document stays untouched); no-registry sentence; + no-detector headline once; `partial` document status; a size document renders byte-identically. **Files Affected** | File | Action | What changes | |---|---|---| -| `plugins/code-metrics/scripts/report.py` | Modify | rollups, sort, summary line, headline | -| `plugins/code-metrics/scripts/test_report.py` | Modify | six cases | -| `plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh` | Modify | pass `--rollup-depth` | -| `plugins/code-metrics/reference/report-schema.md` | Modify | fields, ignore-unknown rule, term | +| `plugins/code-metrics/scripts/report.py` | Modify | rollups, `--root`, status, sort, summary line, headline | +| `plugins/code-metrics/scripts/test_report.py` | Modify | nine cases | +| `plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh` | Modify | pass `--root` and `--rollup-depth` | +| `plugins/code-metrics/reference/report-schema.md` | Modify | fields, status, ignore-unknown rule, term | **Sanity Check:** - `python3 -m unittest plugins/code-metrics/scripts/test_report.py` exits 0. -- `grep -c 'Functions:' <(audit-duplication.sh --all)` prints `0` and - `grep -c '^Files with clones:' <(audit-duplication.sh --all)` prints `1`; the sibling - `audit-size.sh --all | tail -3` still contains `Functions:`. -- `audit-duplication.sh --json --all | jq '[.summary.by_lane[].duplicated_lines]|add == .summary.duplicated_lines'` prints `true`. -- With an empty PATH prefix hiding jscpd, `audit-duplication.sh --all | grep -c 'npm install -g jscpd'` prints `1`. +- `audit-duplication.sh --all | grep -c 'Functions:'` prints `0` and + `audit-duplication.sh --all | grep -c '^Files with clones:'` prints `1`; + `audit-size.sh --all | grep -c 'Functions:'` prints `1`. +- `audit-duplication.sh --json --all | jq '([.summary.by_lane[].duplicated_lines]|add) == .summary.duplicated_lines and .summary.by_directory["."].duplicated_lines == .summary.duplicated_lines'` + prints `true`. +- `PATH= audit-duplication.sh --all | grep -c 'npm install -g jscpd'` prints `1`. ### Phase 5: SKILL.md, README, CHANGELOG, version, dogfood [TODO] -1. `skills/audit-duplication/SKILL.md`: configuration section names the three new keys and both - registry line shapes; "Run it" gains the no-detector instruction (offer the install command to - the user, run it only on confirmation, never silently); "Reading the numbers" states clone - classes, the `partial` row, and the rollup; `## Next` kept before `## Gotchas`; the gotcha - about pairs is rewritten to state clusters. Prose stays in house style (`/ai-slop:audit` on the +1. `skills/audit-duplication/SKILL.md`: configuration section names the three new keys, the + `0`-means-null rule, and both registry line shapes; "Run it" gains the no-detector instruction + (offer the install command to the user, run it only on confirmation, never silently); "Reading + the numbers" states clone classes (byte-aligned copies merge, offset copies stay separate), the + `partial` row and document status, the rollup and its root identity; the pairs gotcha is + rewritten; `## Next` stays before `## Gotchas`. Prose in house style (`/ai-slop:audit` on the file). -2. `README.md`: the audit-duplication row mentions rollups and the cluster line; the known-gaps - bullet's list of prose-restated defaults is checked against what SKILL.md now restates. -3. `CHANGELOG.md`: `## [0.1.9]` with Added / Changed / Fixed entries, one per Brief item; - `.claude-plugin/plugin.json` version 0.1.9. -4. Dogfood: `scripts/affected-tests.sh --run` over the whole diff; `scripts/run-ruff.sh check` +2. `README.md`: the audit-duplication row mentions clone classes, rollups, and the cluster line; + the known-gaps bullet's list of prose-restated defaults is checked against what SKILL.md now + restates. +3. `CHANGELOG.md`: `## [0.1.9]` with Added / Changed / Fixed entries, one per Brief item, including + the offset-copies limitation and the `partial` reading; `.claude-plugin/plugin.json` version 0.1.9. +4. Dogfood: `scripts/affected-tests.sh --run` over the whole diff (exit 0, or exit 3 whose `NOT + RUN` list is Python suites, each then run with `python3 -m pytest`); `scripts/run-ruff.sh check` over the changed Python; `shellcheck` and `shfmt -d` over changed shell; both jscpd majors' - whole-tree runs recorded as distilled numbers in this file's Phase 5 notes (surviving groups, - duplicated lines, exclusions) with no memory-slice paths. + whole-tree runs recorded here as distilled numbers (surviving groups, duplicated lines, + exclusions, partial lanes) with no memory-slice paths. **Files Affected** @@ -399,8 +501,10 @@ Review: code-design - `jq -r .version plugins/code-metrics/.claude-plugin/plugin.json` prints `0.1.9`; `bash scripts/check-changelog-parity.sh --check-bump origin/main` exits 0. -- `scripts/affected-tests.sh --run` exits 0; `scripts/run-ruff.sh check plugins/code-metrics` exits 0. -- `grep -n '^## Next' plugins/code-metrics/skills/audit-duplication/SKILL.md` precedes `^## Gotchas`. +- `scripts/affected-tests.sh --run` exits 0, or exits 3 with every `NOT RUN` line naming a + `test_*.py` that then passes under `python3 -m pytest`; `scripts/run-ruff.sh check plugins/code-metrics scripts/check-code-metrics-config-reference.py` exits 0. +- `grep -n '^## Next' plugins/code-metrics/skills/audit-duplication/SKILL.md` precedes `^## Gotchas`; + `grep -c '5\.1\.2' plugins/code-metrics/reference/collectors.md plugins/code-metrics/scripts/collectors/test_jscpd.py plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh` prints `0` for each. - `markdownlint-cli2` over the changed markdown exits 0. ### Alternatives Considered @@ -409,10 +513,14 @@ Review: code-design |---|---|---| | Detect skips as files-passed minus `statistics.total.sources` | contradicted: `sources` counts token sources that reached detection, reproduced on 4.3.0 and 5.2.0 | jscpd adds a per-file skip list to its JSON report | | Keep jscpd's per-major defaults, detect only | jscpd 4.x users silently lose every file over 1000 lines | jscpd 4.x line reaches end of life | -| Merge pairs inside the jscpd adapter | a second pair-reporting collector would need it again; the rule is about the report | no other collector ever reports pairs and the post-pass is the only consumer | -| A second registry file for cluster lines | two registries for one concept; the plugin's docs already name this file as the shape | a second reader of the registry file cannot be taught to skip cluster lines | +| Any whitespace marks a cluster line | the drift checker's tests protect a registered path containing a space | the registry documents a no-spaces rule for paths | +| A tab or a leading sigil as the cluster marker | less readable than `->` in a hand-edited file; equally unambiguous | a consuming repository has a path containing ` -> ` | +| Merge pairs inside the jscpd adapter | a second pair-reporting collector would need it again; the rule is about the report | no other collector ever reports pairs | +| A second registry file for cluster lines | two registries for one concept; the plugin's docs already name this file as the shape | a second reader of the registry file cannot be taught to skip marked lines | | `by_directory` as direct-parent rows | no documented precedent; a plugin's files would never roll up to the plugin | a consumer needs non-overlapping per-directory sums | -| Stderr prefix as the adapter-to-dispatcher channel | `dispatch.sh` reads stderr only on failure and truncates it; a note file is unambiguous and output-testable | the adapter contract grows a structured stderr protocol for other reasons | +| Stderr prefix as the adapter-to-dispatcher channel | `dispatch.sh` reads stderr only on failure and truncates it; a file is unambiguous and output-testable | the adapter contract grows a structured stderr protocol | +| Strip the install hint out of the run row's reason for the headline | the hint itself contains parentheses and multi-tool lanes concatenate several | never; an additive `hint` field is strictly simpler | +| Exclude `plugins/miro/dist/` through a repo config so no file is skipped here | hides the `partial` reading the change exists to produce; a config is the consuming repo's choice | the repo adopts a `.claude/code-metrics.yaml` for other reasons | ### Test Strategy @@ -425,27 +533,91 @@ fake binary that replays a committed capture; nothing in-process is mocked: `clu argv: `test_jscpd.py`'s existing `argv_log` stub asserts the explicit caps. Red first: each phase writes its failing test against the named boundary, then the change. -Test boundaries (all existing unless marked): `jscpd.py collect` argv and note-file output -(existing CLI, new env vars); `dispatch.sh` run rows (existing); `cluster-clones.py` stdin/stdout -(new script, same document contract as `registry-filter.py`); `registry-filter.py --registry` -(existing); `report.py summarize|resummarize|render` (existing, new `--rollup-depth` argument); -`audit-duplication.sh --json` (existing); `check-cross-plugin-source-drift.sh --check` (existing). -A boundary implementation picks that this list does not name is a deviation logged to -`DEVIATIONS.md` beside this file. - -Edge cases named in the Brief's criteria: overlapping-but-not-identical ranges; two copies in one -directory; all files skipped; no registry configured; a lane whose collector never resolved. -Existing tests updated: `audit-duplication.test.sh` (export assertions, three-copy case), -`test_jscpd.py` (argv), `check-cross-plugin-source-drift.test.sh` (skip case), +Test boundaries (all existing unless marked): `jscpd.py collect` argv and partial-reason output +(existing CLI, new env vars); `dispatch.sh` run rows (existing, new `hint` field); +`cluster-clones.py` stdin/stdout (new script, same document contract as `registry-filter.py`); +`registry-filter.py --registry --root` (existing); `report.py summarize|resummarize|render` +(existing, new `--root` and `--rollup-depth` arguments); `audit-duplication.sh --json` (existing); +`check-cross-plugin-source-drift.sh --check` (existing). A boundary implementation picks that this +list does not name is a deviation logged to `DEVIATIONS.md` beside this file. + +Edge cases named in the Brief's criteria: offset copies; overlapping-but-not-identical ranges; two +copies in one directory; a space-bearing single-token path; a subdirectory cwd; all files skipped; +a `0` cap; no registry configured; a lane whose collector never resolved; a clone-free duplication +run. Existing tests updated: `audit-duplication.test.sh`, `test_jscpd.py`, +`check-cross-plugin-source-drift.test.sh`, `test_registry_filter.py`, `dispatch.test.sh`; `test_setup_apply.py` passes unchanged once the template carries the keys. ### Risks and Mitigations | Risk | Likelihood | Impact | Mitigation | |---|---|---|---| -| jscpd 4.x argv differs from 5.x for a passed cap | Low | Med | both majors verified this session (`--max-lines`, `--max-size` share names and short forms); Phase 1's runtime probe runs both | -| A multi-token registry line breaks a reader not found in pre-flight | Low | High | pre-flight grepped all 12 mentions; the drift-checker test case is the guard; `scripts/affected-tests.sh` selects every suite referencing the registry | -| `by_directory` on a large monorepo makes the JSON heavy | Low | Low | the map holds only directories that contain a group; markdown cuts at depth 2 | -| Union-find merges two genuinely different clones that share one instance | Low | Med | the key requires identical range and equal `lines`, so only the same fragment joins | -| The sort change alters other skills' markdown | Low | Med | sort branch is gated on clone rows; `test_report.py` asserts a size document renders byte-identically | +| jscpd 4.x argv differs from 5.x for a passed cap | Low | Med | both majors verified this session (`--max-lines`, `--max-size` share names; raw byte counts accepted); Phase 1's probe runs both | +| A marked registry line breaks a reader not found in pre-flight | Low | High | pre-flight grepped all 12 mentions; the drift-checker test case is the guard; the registry edit selects every suite referencing it | +| The union-find joins two different fragments | Low | Med | the key requires identical range and equal `lines`; the offset fixture asserts two groups | +| `by_directory` differs by jscpd major | Low | Low | merged instances are path-sorted; the probe compares instance sets, not order | +| The sort or summary change alters other skills' markdown | Low | Med | both branches are gated on clone rows; `test_report.py` asserts a size document renders byte-identically | +| Full-corpus CI run on the registry commit hides an unrelated red | Med | Low | Phase 3 is its own commit; `--check` and its test run locally before the push | | Version bump without changelog entry fails CI | Low | Low | Phase 5 sanity check runs the parity gate | + +## Blast radius + +MEDIUM. About thirty files across one plugin and two repo tooling scripts; one CI gate script and +the registry it reads change, which fans `scripts/affected-tests.sh`'s selection out to most of +the corpus; every change is a revertable commit on a feature branch; the config reference gate, +the changelog parity gate, the drift checker's own test, and the plugin's suites cover it. + +## Stress-test summary + +Both passes ran on the first draft in fresh contexts. The plan reviewer returned 2 CRITICAL, 9 +IMPORTANT, 8 SUGGESTION; `/planning:devils-advocate` returned 1 CRITICAL, 7 HIGH, 6 MEDIUM, 3 LOW, +with probes against both jscpd majors. Every load-bearing finding was verified against the tree +before being applied: the eighteen-instance count; the 1.45mb minified file in the typescript +lane; the non-identical `hook-telemetry-sink.sh` pair with no sync script; the drift test's +annotation-block-per-entry policy and its protected space-bearing path; the zero floor keyed on +`ok`; the cwd rebase of instance paths; the hint glued into the reason string; the whole-path +anchoring of the glob matcher; jscpd's star-shaped pairs with the hub on the last input (4.x) or +the first (5.x); offset copies producing ranges that differ by one line; the fixture-count +assertion in `dispatch.test.sh`; and `affected-tests.sh`'s exit-3 contract. No research-iterate +round was needed: every contested claim was settled by a probe the reviewing agent ran and this +session reproduced. + +## Execution shape + +Fully sequential: 1 → 2 → 3 → 4 → 5. `audit-duplication.sh` is edited in Phases 1, 2, 3 and 4, +`registry-filter.py` in Phases 1 and 3, and Phase 4 renders the shape Phases 2 and 3 produce, so +no two phases are file-disjoint and no parallel wave exists. All-main-session execution. + +| Phase | Surface | Basis | +|---|---|---| +| 1 | main-session | adapter, dispatcher, and config edits interlock; the runtime probe needs the session's scratch jscpd prefixes | +| 2 | main-session | small new script plus captures that must be rewritten by hand to relative names | +| 3 | main-session | a CI gate script and the registry change together and are committed alone | +| 4 | main-session | one shared renderer; judgment on byte-identical sibling output | +| 5 | main-session | prose in house style, release bump, dogfood numbers recorded in this file | + +## Open questions + +None at approval time beyond the gates below. + +## Handoff to implementation + +### User-approval gates + +- The four Brief corrections in the scope-change note at the top of the Brief (instance count, + the `partial` typescript row, the dropped sixth registry line, the byte-aligned merge caveat) + and the `->` marker, which amends the grammar the interview locked; approving this plan approves + them. Any later change to an acceptance criterion stops and asks. + +### Execution shape ([EXEC-SHAPE] tagged) + +- Sequential phases with Phase 3 as its own commit; per-phase sanity checks as written; the + partial-reason file channel; the large explicit `--max-lines` and the bound-plus-one byte cap + passed to jscpd so the adapter's pre-filter is the only gate; fixtures placed outside + `fixtures/sources`; scratch-prefix jscpd installs for probes with `SKIP` when absent. + +### Mechanical work + +- One commit per phase (Phase 3 alone), each carrying its `[DONE]` tag flip in this file; run the + phase's sanity checks before committing; push after each commit. Sequential fallback is not + needed (no parallel wave). At PR time, run `/planning:plan close-out`. diff --git a/docs/topics/code-metrics-duplication-audit/design/design-resolution.md b/docs/topics/code-metrics-duplication-audit/design/design-resolution.md index 90c0fa22ef..1deb6610ed 100644 --- a/docs/topics/code-metrics-duplication-audit/design/design-resolution.md +++ b/docs/topics/code-metrics-duplication-audit/design/design-resolution.md @@ -17,15 +17,17 @@ needs; `/planning:plan` consumes it. ```text line := comment | blank | single | cluster -single := path-within-plugin # unchanged meaning -cluster := canonical-path SP+ member (SP+ member)* # two or more whitespace-separated tokens +single := path-within-plugin # whole line, spaces included; unchanged +cluster := canonical-path " -> " member (SP+ member)* # the arrow is the marker member := repo-relative path | glob (pathglob.py syntax) ``` -A group is excluded by a `cluster` line when every instance's repo-relative path matches the -canonical path or one member, and the instances sit in distinct carrying directories (the prefix in -front of the matched token; the canonical path's carrying directory is its own parent). The -`excluded[]` record keeps `{registry, line, path, instances}` with `path` = the line's text. +Instance paths are normalized to root-relative first (the dispatcher emits them cwd-relative). A +group is excluded by a `cluster` line when every normalized instance equals the canonical path or +matches one member, and the instances' `dirname`s are pairwise distinct (the glob matcher anchors +the whole path, so the single-token line's "prefix before the suffix" rule does not transfer). The +first matching line in file order wins. The `excluded[]` record keeps +`{registry, line, path, instances}` with `path` = the line's text. ### Clone-group row after clustering (unchanged schema, N instances) @@ -48,10 +50,15 @@ pair. Rows from `dupl` and `cpd` pass through untouched (already N-ary). ``` Adapter to dispatcher channel: the adapter writes the skip note to the path in -`CODE_METRICS_RUN_NOTE_FILE` (set by `dispatch.sh` per lane/measure/tool); when the file is -non-empty after a successful collect, `dispatch.sh` writes the run row as `partial` with that text. -When the pre-filter leaves zero files, the adapter writes the note and exits 0 without invoking the -tool; the row is `partial` and no `exit 3` occurs. +`CODE_METRICS_PARTIAL_REASON_FILE` (set by `dispatch.sh` per lane/measure/tool, in a work dir +that is fresh per run); when the file is non-empty after a successful collect, `dispatch.sh` writes +the run row as `partial` with that text. When the variable is unset the note goes to stderr and is +never a failure. When the pre-filter leaves zero files, the adapter writes the note and exits 0 +without invoking the tool; the row is `partial` and no `exit 3` occurs. A failed probe's run row +carries the adapter's install hint in an additive `hint` field beside the unchanged `reason`. + +The merged clone-group row's instances are sorted by `(file, start_line)`, so the first instance +is the same whichever jscpd major produced the pairs (4.x hubs on the last input, 5.x on the first). ### Summary additions (additive `code-metrics/v1`) From b0234f241cba130c5b516bcd9edc16eae4904b5a Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 15:58:34 +0000 Subject: [PATCH 05/15] feat(code-metrics): explicit duplication caps with a partial run row Phase 1 of the duplication audit plan (docs/topics/code-metrics-duplication-audit/PLAN.md). - Add duplication.max_lines (null), duplication.max_size (1mb), and duplication.rollup_depth (2) to the bundled defaults, the config reference, and the setup template. - audit-duplication.sh exports CODE_METRICS_DUP_MAX_LINES and CODE_METRICS_DUP_MAX_SIZE beside the existing tunables; a null or 0 cap exports empty. - The jscpd adapter applies both caps itself before jscpd runs (stat size, binary newline count only when a line cap is set) and passes jscpd one above its own bound, because jscpd 4.x and 5.x disagree on the --max-size/--max-lines defaults and on what 0 means, and neither names a skipped file. Every skip is one line to CODE_METRICS_PARTIAL_REASON_FILE, or stderr when unset; zero files left is a zero measurement, not a failure. Both majors translate. - dispatch.sh gives each collect a partial-reason file and writes the run row as `partial` with that line when it is non-empty; every run row gains an additive `hint` field carrying the first install hint a failed probe produced. - registry-filter.py --zero-floor counts a `partial` duplication row as measured. - collectors.md pins jscpd 5.2.0 and records the 4.3.0 maintenance line and the caps semantics; report-schema.md documents `hint`. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01QShQwS5yDYYToBmofdEY5N --- .../code-metrics-duplication-audit/PLAN.md | 2 +- plugins/code-metrics/reference/collectors.md | 4 +- plugins/code-metrics/reference/config.md | 3 + .../code-metrics/reference/report-schema.md | 14 +- .../code-metrics/scripts/collectors/jscpd.py | 185 ++++++++++++++++-- .../scripts/collectors/test_jscpd.py | 147 +++++++++++++- .../code-metrics/scripts/config-defaults.json | 5 +- plugins/code-metrics/scripts/dispatch.sh | 29 ++- plugins/code-metrics/scripts/dispatch.test.sh | 26 ++- .../scripts/audit-duplication.sh | 24 ++- .../scripts/audit-duplication.test.sh | 6 +- .../scripts/registry-filter.py | 3 +- .../scripts/test_registry_filter.py | 16 ++ .../setup/templates/config-template.yaml | 3 + 14 files changed, 429 insertions(+), 38 deletions(-) diff --git a/docs/topics/code-metrics-duplication-audit/PLAN.md b/docs/topics/code-metrics-duplication-audit/PLAN.md index 8fdbc0a938..764e6ae010 100644 --- a/docs/topics/code-metrics-duplication-audit/PLAN.md +++ b/docs/topics/code-metrics-duplication-audit/PLAN.md @@ -224,7 +224,7 @@ into two scratch prefixes outside the repository and prepends the wanted `node_m `PATH` per probe. A probe whose major is absent prints `SKIP` and is not a failure, the way `audit-duplication.test.sh`'s real-cluster case already does. -### Phase 1: Explicit caps, adapter pre-filter, partial run row [TODO] +### Phase 1: Explicit caps, adapter pre-filter, partial run row [DONE] Review: code-design diff --git a/plugins/code-metrics/reference/collectors.md b/plugins/code-metrics/reference/collectors.md index 30bbbea5c8..3ba6449153 100644 --- a/plugins/code-metrics/reference/collectors.md +++ b/plugins/code-metrics/reference/collectors.md @@ -33,7 +33,9 @@ built them could not run the tool; the first live run is that row's recheck trig | Tool or format | Lane(s) | Measure | Claim the adapter relies on | Basis | Verified | Recheck trigger | |---|---|---|---|---|---|---| -| `jscpd` 5.1.2 | every lane | `duplication` | v5 is a Rust binary that only writes `/jscpd-report.json`, never stdout, so the adapter runs `--reporters json --output ` and prints the file itself; `duplicates[]` carries `firstFile`/`secondFile` with `start`, `end`, plus `lines` and `tokens`; `--absolute` is required because names are otherwise relative to the common ancestor of the inputs, which collapses two vendored copies sharing a basename; jscpd 4 wrote a different document under the same name | github.com/kucherenko/jscpd, run in this repository (the capture is `scripts/fixtures/tool-output/jscpd.json`) | 2026-09-05 | a jscpd major release, or a change to the report filename or the `duplicates[]` shape | +| `jscpd` 5.2.0 | every lane | `duplication` | v5 is a Rust binary that only writes `/jscpd-report.json`, never stdout, so the adapter runs `--reporters json --output ` and prints the file itself; `duplicates[]` carries `firstFile`/`secondFile` with `start`, `end`, plus `lines` and `tokens`, and 5.2.0 adds a per-duplicate `kind` (`exact`) the adapter does not read; `--absolute` is required because names are otherwise relative to the common ancestor of the inputs, which collapses two vendored copies sharing a basename; `statistics.total.sources` counts token sources, not files, so it is not a skipped-file signal | github.com/kucherenko/jscpd, run in this repository (the capture is `scripts/fixtures/tool-output/jscpd.json`, a 5.1.2 capture whose keys 5.2.0 reproduces) | 2026-09-11 | a jscpd major release, or a change to the report filename or the `duplicates[]` shape | +| `jscpd` 4.3.0 (the 4.x maintenance line) | every lane | `duplication` | v4 is a Node program that writes the same `/jscpd-report.json` with the same `firstFile`/`secondFile`, `start`, `end`, `lines`, and `tokens` keys, so one adapter translates both majors; it tokenizes differently from v5, so a clone count can differ between the majors on the same input, and the plugin never compares counts across a major boundary | github.com/kucherenko/jscpd (`npm install jscpd@4`), run in this repository over the same cluster fixture | 2026-09-11 | a 4.x release that changes the report shape, or the 4.x line being retired upstream | +| `jscpd` size and line caps (both majors) | every lane | `duplication` | the adapter applies `duplication.max_size` and `duplication.max_lines` itself, before jscpd runs, and passes jscpd one above its own bound (or `1000000` lines and `1099511627776` bytes when there is no cap), because the majors disagree on the flags: 4.x defaults to 100kb and 1000 lines and reads `--max-lines 0` as that default, 5.x defaults to 1mb with no line cap, both read `--max-size 0` as skip every file, and neither names a skipped file in the report; jscpd's size grammar is binary (`1kb` is 1,024 bytes, `1mb` is 1,048,576) and the adapter uses the same multipliers | github.com/kucherenko/jscpd, both majors run in this repository with `--max-size` and `--max-lines` set to `0`, the default, and one byte or line below a fixture file's size | 2026-09-11 | either major changes a `--max-size`/`--max-lines` default or the meaning of `0`, or a report gains a skipped-file list | | PMD CPD 7.27.0 | typescript, python, go, dotnet | `duplication` | `pmd cpd --minimum-tokens N --format xml --language --file-list ` (one path per line) prints a namespaced `pmd-cpd` document whose `duplication` elements carry `lines` and `tokens` with one `file` child per instance (`path`, `line`, `endline`); CPD has no JSON reporter, no minimum-lines option, no ignore-glob option, and no Bash or shell language; exit 4 means duplications were found, not that the run failed | docs.pmd-code.org CPD user documentation, CLI reference, and report formats; the adapter and its fixture are unverified against a live run | 2026-09-05 | a PMD 8 release, a JSON reporter, a shell CPD language, or the first live run of this adapter | | `dupl` v1.1.0 | go | `duplication` | the default text printer emits `found clones:` per group, then an indented `:,` line per instance, then a total footer; `-plumbing` is pairwise and loses groups of three or more, so the text printer is parsed; `-t` is a token threshold with no line equivalent; dupl reports no token count and ships no version flag | github.com/mibk/dupl `printer/text.go` and `main.go`; the adapter and its fixture are unverified against a live run | 2026-09-05 | a dupl release that changes the printer, adds a version flag, or adds a token count | diff --git a/plugins/code-metrics/reference/config.md b/plugins/code-metrics/reference/config.md index bdeea388f4..0f145f0e87 100644 --- a/plugins/code-metrics/reference/config.md +++ b/plugins/code-metrics/reference/config.md @@ -69,6 +69,9 @@ The third column is written by hand and is not derived from anything. A row whos | `duplication.min_lines` | `5` | Passed to the clone collector | | `duplication.ignore` | `[]` | Collector ignore globs | | `duplication.registries` | `[]` | Sanctioned-replication registries (one path-within-plugin per line); a clone whose every instance sits at a listed path is excluded, not suppressed | +| `duplication.max_lines` | `null` | A file with more lines is left out of the clone scan and named in the lane's `partial` run row; `null` or `0` means no line cap, which is what jscpd 5, PMD CPD, and SonarQube ship. A number is a plugin-local guard, not an upstream convention | +| `duplication.max_size` | `1mb` | A file larger than this is left out and named the same way; `0` means no cap. jscpd 5.0.7 sets 1mb as its parser guard and SonarJS 1000kb for generated code. `kb` and `mb` are binary (1mb is 1,048,576 bytes); a CRLF checkout counts one more byte per line | +| `duplication.rollup_depth` | `2` | Directory depth to which the markdown report lists per-directory rollup rows; the JSON carries every directory | | `coverage.artifacts` | `[]` | Explicit coverage artifact paths; empty means auto-discover. An explicitly named path that does not exist is a usage error | | `coverage.path_prefix_strip` | `[]` | Prefixes removed from artifact paths before the join with source paths (compiled-output layouts) | | `coverage.reference` | `null` | No default bar; ISO/IEC 25023 files coverage under Reliability and sets no value | diff --git a/plugins/code-metrics/reference/report-schema.md b/plugins/code-metrics/reference/report-schema.md index e10dba42e3..bfc1144429 100644 --- a/plugins/code-metrics/reference/report-schema.md +++ b/plugins/code-metrics/reference/report-schema.md @@ -29,14 +29,18 @@ count as one even though only the first reports where it begins. ## `run[]` rows `lane`, `measure`, `collector` (the tool and version that produced the rows, or `null`), `status` -(`ok`, `partial`, `unavailable`, `not-applicable`, `deferred`), `reason` (`null` only when `ok`). A -run whose scope holds no measurable file carries one row `*/*` with status `not-applicable` and the -reason `no measurable files in scope`. +(`ok`, `partial`, `unavailable`, `not-applicable`, `deferred`), `reason` (`null` only when `ok`), +`hint` (the first install hint a failed probe produced for the row, or `null`; it is kept apart +from the prose reason so a renderer can print it once without parsing it back out). A run whose +scope holds no measurable file carries one row `*/*` with status `not-applicable` and the reason +`no measurable files in scope`. `partial` means the row produced measurements for some of what it implied and not the rest, which `audit-coverage` emits when an artifact covers only some of a lane's scope files, and again when it -left a function unjoined, naming those functions in the reason. It counts as having produced rows, -so such a run is `partial` rather than `empty`, and it withholds `complete`, so a document can never +left a function unjoined, naming those functions in the reason, and which `audit-duplication` emits +when a `duplication.max_size` or `duplication.max_lines` cap left files out of the clone scan, +naming the count and the largest skipped file in the reason. It counts as having produced rows, so +such a run is `partial` rather than `empty`, and it withholds `complete`, so a document can never read as complete while one of its own rows says `N of M`. ## `measures[]` rows diff --git a/plugins/code-metrics/scripts/collectors/jscpd.py b/plugins/code-metrics/scripts/collectors/jscpd.py index dd566cdb1c..18894000a8 100755 --- a/plugins/code-metrics/scripts/collectors/jscpd.py +++ b/plugins/code-metrics/scripts/collectors/jscpd.py @@ -4,15 +4,20 @@ Adapter contract (design/contracts.md section 3): `probe`, `measures`, `collect ...`, `install_hint`. -jscpd 5 is a Rust binary that only writes its report to a file, so `collect` -runs it with `--reporters json --output `, reads -`/jscpd-report.json`, prints the translated rows, and deletes the -temporary directory (probed 2026-09-05 against jscpd 5.1.2). `--absolute` is -passed because jscpd otherwise names files relative to the common ancestor of -its inputs, which collapses two vendored copies that share a basename into one -indistinguishable name; the absolute paths are made relative to the working -directory here. jscpd 4 wrote a different document under the same name and is -not translated by this file. +jscpd only writes its report to a file, so `collect` runs it with +`--reporters json --output `, reads `/jscpd-report.json`, +prints the translated rows, and deletes the temporary directory. Both +maintained majors are translated: the 4.x line (Node) and the 5.x line (Rust) +write `duplicates[]` entries with `firstFile`/`secondFile` under the same +report name, and this adapter reads only those keys plus `lines` and `tokens` +(verified 2026-09-11 against jscpd 4.3.0 and 5.2.0; recheck when a major above +5 ships). 5.2.0 adds a per-duplicate `kind` (`exact`) that is not read, and +the two majors tokenize differently, so a clone count can differ between them +on the same input. +`--absolute` is passed because jscpd otherwise names files relative to the +common ancestor of its inputs, which collapses two vendored copies that share +a basename into one indistinguishable name; the absolute paths are made +relative to the working directory here. Each duplicate becomes one clone-group row: `file` and `function` are null, `instances[]` carries every copy with its line range, and `values` carries @@ -22,6 +27,23 @@ CODE_METRICS_DUP_MIN_TOKENS jscpd --min-tokens (default 50) CODE_METRICS_DUP_MIN_LINES jscpd --min-lines (default 5) CODE_METRICS_DUP_IGNORE jscpd --ignore, comma-separated globs (default none) + CODE_METRICS_DUP_MAX_SIZE files larger than this are skipped (default 1mb; + empty or 0 means no cap; kb/mb/gb are binary) + CODE_METRICS_DUP_MAX_LINES files with more lines are skipped (default none; + empty or 0 means no cap) + +The caps are applied HERE, before jscpd runs, and not delegated to jscpd's +own `--max-size`/`--max-lines`: the two majors disagree on what those flags +default to (4.x caps at 1000 lines and 100kb, 5.x at 1mb and no line cap), +on what `0` means (4.x reads `--max-lines 0` as the default and `--max-size 0` +as "skip everything"), and neither names a skipped file in the report, so a +file left out of the scan would be invisible. jscpd is passed one more than +the adapter's own bound (or a bound no real file reaches when there is no +cap) so the pre-filter is the only gate on either major. Every skip is +reported as one line to the file named by CODE_METRICS_PARTIAL_REASON_FILE +(the dispatcher's channel for a `partial` run row), or to stderr when that +variable is unset. `statistics.total.sources` in the jscpd report counts token +sources, not files, and is not read for this purpose. jscpd's own exit code is not read: it exits non-zero when a `--threshold` or `--exit-code` run finds clones, and this adapter passes neither, so the report @@ -44,6 +66,13 @@ REPORT_BASENAME = "jscpd-report.json" DEFAULT_MIN_TOKENS = "50" DEFAULT_MIN_LINES = "5" +DEFAULT_MAX_SIZE = "1mb" +# Passed to jscpd when the adapter applies no cap of its own: bounds no file +# that survives the pre-filter reaches, valid on both majors (`0` is not). +NO_LINE_CAP = 1_000_000 +NO_SIZE_CAP = 1 << 40 +_SIZE_UNITS = {"": 1, "b": 1, "kb": 1024, "mb": 1024**2, "gb": 1024**3} +_SIZE_RE = re.compile(r"^(\d+(?:\.\d+)?)\s*([kmg]?b)?$") def _normalize(path: str) -> str: @@ -85,6 +114,114 @@ def probe() -> int: return 0 +def parse_size(text: str) -> int | None: + """Bytes for a size such as `1mb`, `100kb`, `2048`; None for no cap. + + Units are binary (1kb = 1024 bytes), the multiplier jscpd's own grammar + uses. Empty and `0` mean no cap. Anything else raises ValueError. + """ + text = (text or "").strip().lower() + if not text: + return None + match = _SIZE_RE.match(text) + if not match: + raise ValueError(f"not a size: {text!r} (expected e.g. 1mb, 100kb, 2048)") + value = int(float(match.group(1)) * _SIZE_UNITS[match.group(2) or ""]) + return value if value > 0 else None + + +def parse_lines(text: str) -> int | None: + """A positive line cap, or None for no cap (empty, 0, or negative).""" + text = (text or "").strip() + if not text: + return None + try: + value = int(text) + except ValueError as exc: + raise ValueError(f"not a line count: {text!r}") from exc + return value if value > 0 else None + + +def count_lines(path: str) -> int: + """Newline count, plus one for a final line without a newline.""" + lines = 0 + last = b"\n" + with open(path, "rb") as handle: + for chunk in iter(lambda: handle.read(1 << 16), b""): + lines += chunk.count(b"\n") + last = chunk[-1:] + if last != b"\n": + lines += 1 + return lines + + +def prefilter( + files: list[str], size_cap: int | None, line_cap: int | None +) -> tuple[list[str], list[tuple[str, int, int | None]]]: + """Split `files` into the ones jscpd scans and the ones a cap skips. + + A skipped entry is `(path, bytes, lines)`; `lines` is None when no line cap + is set, because counting lines reads the whole file and the size cap + needs only a stat. A file that cannot be stat'ed is kept, so jscpd (and its + own error message) decides what to do with it. + """ + kept: list[str] = [] + skipped: list[tuple[str, int, int | None]] = [] + for path in files: + try: + size = os.stat(path).st_size + except OSError: + kept.append(path) + continue + lines: int | None = None + if size_cap is not None and size > size_cap: + skipped.append((path, size, None)) + continue + if line_cap is not None: + try: + lines = count_lines(path) + except OSError: + kept.append(path) + continue + if lines > line_cap: + skipped.append((path, size, lines)) + continue + kept.append(path) + return kept, skipped + + +def report_skips( + skipped: list[tuple[str, int, int | None]], + total: int, + size_text: str, + lines_text: str, +) -> None: + """One line naming how many files a cap left out, and the largest one.""" + if not skipped: + return + largest = max(skipped, key=lambda entry: entry[1]) + path, size, lines = largest + if lines is None: + try: + lines = count_lines(path) + except OSError: + lines = 0 + reason = ( + f"{len(skipped)} of {total} files skipped by duplication.max_size " + f"{size_text or 'none'} / max_lines {lines_text or 'none'}; " + f"largest: {_normalize(path)} ({size} bytes, {lines} lines)" + ) + target = os.environ.get("CODE_METRICS_PARTIAL_REASON_FILE") or "" + if target: + try: + with open(target, "w", encoding="utf-8") as handle: + handle.write(reason + "\n") + return + except OSError as exc: + print(f"jscpd.py: cannot write {target}: {exc}", file=sys.stderr) + print(f"jscpd.py: {reason}", file=sys.stderr) + + def _instance(entry: dict) -> dict: return { "file": _normalize(str(entry.get("name", ""))), @@ -121,7 +258,13 @@ def translate(raw: str, lane: str) -> list[dict]: return rows -def _command(exe: str, output: str, files: list[str]) -> list[str]: +def _command( + exe: str, + output: str, + files: list[str], + size_cap: int | None = None, + line_cap: int | None = None, +) -> list[str]: command = [ exe, "--reporters", @@ -132,6 +275,12 @@ def _command(exe: str, output: str, files: list[str]) -> list[str]: os.environ.get("CODE_METRICS_DUP_MIN_TOKENS") or DEFAULT_MIN_TOKENS, "--min-lines", os.environ.get("CODE_METRICS_DUP_MIN_LINES") or DEFAULT_MIN_LINES, + # One above the adapter's own bound: the pre-filter already removed + # every file over it, so jscpd's gate never fires on either major. + "--max-size", + str(size_cap + 1 if size_cap is not None else NO_SIZE_CAP), + "--max-lines", + str(line_cap + 1 if line_cap is not None else NO_LINE_CAP), "--absolute", "--silent", ] @@ -149,10 +298,24 @@ def collect(lane: str, measure: str, files: list[str]) -> int: if not exe: print("jscpd not on PATH or in ./node_modules/.bin", file=sys.stderr) return 3 + size_text = os.environ.get("CODE_METRICS_DUP_MAX_SIZE", DEFAULT_MAX_SIZE) + lines_text = os.environ.get("CODE_METRICS_DUP_MAX_LINES", "") + try: + size_cap = parse_size(size_text) + line_cap = parse_lines(lines_text) + except ValueError as exc: + print(f"jscpd.py: {exc}", file=sys.stderr) + return 2 + kept, skipped = prefilter(files, size_cap, line_cap) + report_skips(skipped, len(files), size_text.strip(), lines_text.strip()) + if not kept: + # Nothing left to scan is a measurement of zero, not a failure; the + # skip line above says what was left out. + return 0 output = tempfile.mkdtemp(prefix="code-metrics-jscpd-") try: result = subprocess.run( - _command(exe, output, files), + _command(exe, output, kept, size_cap, line_cap), capture_output=True, text=True, check=False, diff --git a/plugins/code-metrics/scripts/collectors/test_jscpd.py b/plugins/code-metrics/scripts/collectors/test_jscpd.py index e057a2b8ae..a44e14d49d 100755 --- a/plugins/code-metrics/scripts/collectors/test_jscpd.py +++ b/plugins/code-metrics/scripts/collectors/test_jscpd.py @@ -7,7 +7,10 @@ `--output` directory the adapter passes, the way jscpd 5 writes its own report (design T13; no executable is committed). The capture came from a live jscpd 5.1.2 run over the two-copy cluster under -fixtures/sources/cluster/{alpha,beta}/shared/shared-utils.sh. +fixtures/sources/cluster/{alpha,beta}/shared/shared-utils.sh; 5.2.0 writes the +same document plus a per-duplicate `kind` the adapter does not read, and 4.3.0 +writes the same keys the adapter does read, so one capture stands in for both +majors and the stub only varies the version line. """ from __future__ import annotations @@ -33,7 +36,7 @@ def make_stub( directory: Path, - version_line: str = "jscpd 5.1.2", + version_line: str = "jscpd 5.2.0", capture: Path = CAPTURE, exit_code: int = 0, argv_log: Path | None = None, @@ -90,7 +93,7 @@ def test_probe_prints_the_version_when_the_stub_resolves(self) -> None: with tempfile.TemporaryDirectory() as tmp: make_stub(Path(tmp)) result = run("probe", path_prefix=Path(tmp)) - self.assertEqual((result.returncode, result.stdout.strip()), (0, "5.1.2")) + self.assertEqual((result.returncode, result.stdout.strip()), (0, "5.2.0")) def test_collect_translates_the_capture_into_one_clone_group(self) -> None: with tempfile.TemporaryDirectory() as tmp: @@ -185,7 +188,7 @@ def test_no_report_file_is_exit_3_with_the_tool_stderr(self) -> None: stub = Path(tmp) / "jscpd" stub.write_text( "#!/usr/bin/env bash\n" - 'if [[ "${1:-}" == "--version" ]]; then printf \'jscpd 5.1.2\\n\'; exit 0; fi\n' + 'if [[ "${1:-}" == "--version" ]]; then printf \'jscpd 5.2.0\\n\'; exit 0; fi\n' "printf 'jscpd: unsupported format\\n' >&2\n" "exit 1\n", encoding="utf-8", @@ -204,6 +207,142 @@ def test_the_temporary_output_directory_is_removed(self) -> None: run("collect", "bash", "duplication", ALPHA, BETA, path_prefix=Path(tmp)) self.assertEqual(set(glob.glob(pattern)) - before, set()) + def test_explicit_caps_reach_the_command_line_on_both_majors(self) -> None: + for version in ("jscpd 4.3.0", "jscpd 5.2.0"): + with tempfile.TemporaryDirectory() as tmp: + log = Path(tmp) / "argv.log" + make_stub(Path(tmp), version_line=version, argv_log=log) + result = run( + "collect", + "bash", + "duplication", + ALPHA, + BETA, + path_prefix=Path(tmp), + env_extra={ + "CODE_METRICS_DUP_MAX_SIZE": "1mb", + "CODE_METRICS_DUP_MAX_LINES": "", + }, + ) + self.assertEqual(result.returncode, 0, result.stderr) + argv = log.read_text(encoding="utf-8") + # One byte above the adapter's own bound, so the pre-filter is + # the only gate on either major. + self.assertIn("--max-size 1048577", argv, version) + self.assertIn("--max-lines 1000000", argv, version) + + def test_a_zero_cap_means_no_cap_and_is_never_passed(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + log = Path(tmp) / "argv.log" + make_stub(Path(tmp), version_line="jscpd 4.3.0", argv_log=log) + result = run( + "collect", + "bash", + "duplication", + ALPHA, + path_prefix=Path(tmp), + env_extra={ + "CODE_METRICS_DUP_MAX_SIZE": "0", + "CODE_METRICS_DUP_MAX_LINES": "0", + }, + ) + self.assertEqual(result.returncode, 0, result.stderr) + argv = log.read_text(encoding="utf-8") + self.assertNotIn("--max-size 0 ", argv + " ") + self.assertNotIn("--max-lines 0 ", argv + " ") + self.assertIn("--max-lines 1000000", argv) + self.assertIn("--max-size 1099511627776", argv) + + def test_the_size_grammar_uses_binary_multipliers(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + log = Path(tmp) / "argv.log" + make_stub(Path(tmp), argv_log=log) + result = run( + "collect", + "bash", + "duplication", + ALPHA, + path_prefix=Path(tmp), + env_extra={ + "CODE_METRICS_DUP_MAX_SIZE": "2kb", + "CODE_METRICS_DUP_MAX_LINES": "44", + }, + ) + self.assertEqual(result.returncode, 0, result.stderr) + argv = log.read_text(encoding="utf-8") + self.assertIn("--max-size 2049", argv) + self.assertIn("--max-lines 45", argv) + + def test_files_over_the_cap_are_skipped_and_the_reason_is_recorded(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + log = Path(tmp) / "argv.log" + note = Path(tmp) / "partial" + big = Path(tmp) / "big.sh" + big.write_text("echo line\n" * 100, encoding="utf-8") + make_stub(Path(tmp), argv_log=log) + result = run( + "collect", + "bash", + "duplication", + ALPHA, + str(big), + path_prefix=Path(tmp), + env_extra={ + "CODE_METRICS_DUP_MAX_LINES": "50", + "CODE_METRICS_PARTIAL_REASON_FILE": str(note), + }, + ) + self.assertEqual(result.returncode, 0, result.stderr) + argv = log.read_text(encoding="utf-8") + self.assertIn(ALPHA, argv) + self.assertNotIn("big.sh", argv) + self.assertRegex( + note.read_text(encoding="utf-8").strip(), + r"^1 of 2 files skipped by duplication\.max_size 1mb / max_lines 50; " + r"largest: .*big\.sh \(\d+ bytes, 100 lines\)$", + ) + + def test_all_files_skipped_returns_zero_without_invoking_jscpd(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + log = Path(tmp) / "argv.log" + note = Path(tmp) / "partial" + make_stub(Path(tmp), argv_log=log) + result = run( + "collect", + "bash", + "duplication", + ALPHA, + BETA, + path_prefix=Path(tmp), + env_extra={ + "CODE_METRICS_DUP_MAX_SIZE": "10", + "CODE_METRICS_PARTIAL_REASON_FILE": str(note), + }, + ) + self.assertEqual((result.returncode, result.stdout), (0, ""), result.stderr) + self.assertFalse(log.exists(), "jscpd was invoked with no files") + self.assertTrue( + note.read_text(encoding="utf-8").startswith( + "2 of 2 files skipped by duplication.max_size 10 / max_lines none" + ), + note.read_text(encoding="utf-8"), + ) + + def test_the_skip_reason_goes_to_stderr_when_no_reason_file_is_set(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + make_stub(Path(tmp)) + result = run( + "collect", + "bash", + "duplication", + ALPHA, + BETA, + path_prefix=Path(tmp), + env_extra={"CODE_METRICS_DUP_MAX_SIZE": "10"}, + ) + self.assertEqual(result.returncode, 0) + self.assertIn("2 of 2 files skipped", result.stderr) + def test_other_verbs(self) -> None: self.assertEqual(run("measures").stdout.strip(), "*/duplication") self.assertIn("kucherenko/jscpd", run("install_hint").stdout) diff --git a/plugins/code-metrics/scripts/config-defaults.json b/plugins/code-metrics/scripts/config-defaults.json index b9315b1b94..fe739e0063 100644 --- a/plugins/code-metrics/scripts/config-defaults.json +++ b/plugins/code-metrics/scripts/config-defaults.json @@ -18,7 +18,10 @@ "min_tokens": 50, "min_lines": 5, "ignore": [], - "registries": [] + "registries": [], + "max_lines": null, + "max_size": "1mb", + "rollup_depth": 2 }, "coverage": { "artifacts": [], diff --git a/plugins/code-metrics/scripts/dispatch.sh b/plugins/code-metrics/scripts/dispatch.sh index d144e076bd..4925eb3ff0 100755 --- a/plugins/code-metrics/scripts/dispatch.sh +++ b/plugins/code-metrics/scripts/dispatch.sh @@ -464,12 +464,16 @@ json_str() { } run_row() { - # run_row - local collector reason + # run_row [] + # `hint` is the first install hint a failed probe produced for the row, kept + # apart from the prose reason so a renderer can print it once without + # parsing it back out. + local collector reason hint if [[ -n "$3" ]]; then collector="$(json_str "$3")"; else collector=null; fi if [[ -n "$5" ]]; then reason="$(json_str "$5")"; else reason=null; fi - printf '{"lane": %s, "measure": %s, "collector": %s, "status": %s, "reason": %s}\n' \ - "$(json_str "$1")" "$(json_str "$2")" "$collector" "$(json_str "$4")" "$reason" >>"$RUN" + if [[ -n "${6:-}" ]]; then hint="$(json_str "$6")"; else hint=null; fi + printf '{"lane": %s, "measure": %s, "collector": %s, "status": %s, "reason": %s, "hint": %s}\n' \ + "$(json_str "$1")" "$(json_str "$2")" "$collector" "$(json_str "$4")" "$reason" "$hint" >>"$RUN" } IFS=',' read -r -a MEASURE_LIST <<<"$MEASURES" @@ -484,6 +488,7 @@ for lane in "${LANES[@]}"; do for measure in "${MEASURE_LIST[@]}"; do resolved=0 reasons="" + first_hint="" while IFS=$'\t' read -r tool note; do [[ -n "$tool" ]] || continue case "$tool" in @@ -517,15 +522,25 @@ for lane in "${LANES[@]}"; do why="$(tr '\n' ' ' <"$probe_err" | cut -c1-200)" why="${why% }" reasons+="${reasons:+; }$tool: ${why:-not found}${hint:+ ($hint)}" + [[ -n "$first_hint" || -z "$hint" ]] || first_hint="$hint" continue fi errf="$WORK/err.$lane.$measure.$tool" outf="$WORK/out.$lane.$measure.$tool" - "${PY[@]}" "$adapter" collect "$lane" "$measure" "${lane_files[@]}" >"$outf" 2>"$errf" + # A collector that leaves some of its inputs out (a cap it applies + # itself) writes one line here; a successful collect with a non-empty + # file is a `partial` row carrying that line, never a silent `ok`. + partialf="$WORK/partial.$lane.$measure.$tool" + CODE_METRICS_PARTIAL_REASON_FILE="$partialf" \ + "${PY[@]}" "$adapter" collect "$lane" "$measure" "${lane_files[@]}" >"$outf" 2>"$errf" rc=$? if [[ $rc -eq 0 ]]; then cat "$outf" >>"$ROWS" - run_row "$lane" "$measure" "$tool $version" ok '' + if [[ -s "$partialf" ]]; then + run_row "$lane" "$measure" "$tool $version" partial "$(head -n 1 "$partialf")" + else + run_row "$lane" "$measure" "$tool $version" ok '' + fi else COLLECT_FAILED=1 run_row "$lane" "$measure" "$tool $version" unavailable "collect failed (exit $rc): $(tr '\n' ' ' <"$errf" | cut -c1-500)" @@ -534,7 +549,7 @@ for lane in "${LANES[@]}"; do break done < <(ladder_tools "$lane" "$measure") if [[ $resolved -eq 0 ]]; then - run_row "$lane" "$measure" '' unavailable "${reasons:-no ladder entry for $lane/$measure}" + run_row "$lane" "$measure" '' unavailable "${reasons:-no ladder entry for $lane/$measure}" "$first_hint" fi done done diff --git a/plugins/code-metrics/scripts/dispatch.test.sh b/plugins/code-metrics/scripts/dispatch.test.sh index 75d7dad688..a180465c31 100755 --- a/plugins/code-metrics/scripts/dispatch.test.sh +++ b/plugins/code-metrics/scripts/dispatch.test.sh @@ -101,6 +101,8 @@ assert_doc "status empty and every row unavailable with a reason" "$out" \ 'd["status"]=="empty" and d["run"] and all(r["status"]!="ok" and r["reason"] for r in d["run"]) and len(d["unavailable"])==5 and d["measures"]==[]' assert_doc "the reason names both rungs and the install hint" "$out" \ '"scc: scc not on PATH" in d["run"][0]["reason"] and "line-counter: disabled by CODE_METRICS_DISABLE_BUNDLED" in d["run"][0]["reason"] and "boyter/scc" in d["run"][0]["reason"]' +assert_doc "an unavailable row also carries the first install hint as its own field" "$out" \ + 'd["run"][0]["hint"] and "boyter/scc" in d["run"][0]["hint"]' # 4. A ladder row whose adapter does not exist is reported, not skipped. ladder="$(mktemp)" @@ -131,7 +133,7 @@ assert_eq "missing skill name exits 2" 2 "$?" empty_dir="$(mktemp -d)" out="$(PATH="$EMPTY_PATH" bash "$SCRIPT" audit-size --measures file_lines --all "$empty_dir")" assert_doc "empty scope yields one not-applicable row" "$out" \ - 'd["status"]=="empty" and d["scope"]["files"]==0 and d["run"]==[{"lane":"*","measure":"*","collector":None,"status":"not-applicable","reason":"no measurable files in scope"}]' + 'd["status"]=="empty" and d["scope"]["files"]==0 and d["run"]==[{"lane":"*","measure":"*","collector":None,"status":"not-applicable","reason":"no measurable files in scope","hint":None}]' rmdir "$empty_dir" # 8. A collector that probes but fails in collect: exit 3, row unavailable. @@ -413,5 +415,27 @@ case "$err" in esac rm -rf "$repo" "$home" +# 18. A collector that leaves inputs out reports the lane as partial with its +# reason, through the partial-reason file the dispatcher hands every +# collect. The jscpd adapter skips both cluster copies under a 100-byte +# cap, so the fake `jscpd` (probe only; never reached for collect) has +# nothing to replay. +skipper="$(mktemp -d)" +cat >"$skipper/jscpd" <<'EOF' +#!/usr/bin/env bash +if [[ "${1:-}" == "--version" ]]; then printf 'jscpd 5.2.0\n'; exit 0; fi +printf 'jscpd should not have been invoked\n' >&2 +exit 1 +EOF +chmod +x "$skipper/jscpd" +out="$(PATH="$skipper:$EMPTY_PATH" CODE_METRICS_DUP_MAX_SIZE=100 bash "$SCRIPT" audit-duplication --measures duplication --all "$SOURCES/cluster")" +rc=$? +assert_eq "a run whose inputs were all skipped exits 0" 0 "$rc" +assert_doc "a skipped input makes the lane row partial with the reason" "$out" \ + 'any(r["lane"]=="bash" and r["measure"]=="duplication" and r["status"]=="partial" and r["reason"].startswith("2 of 2 files skipped by duplication.max_size 100") for r in d["run"])' +assert_doc "a skipped input leaves the row's hint null" "$out" \ + 'all(r.get("hint") is None for r in d["run"])' +rm -rf "$skipper" + printf '%d cases, %d failed\n' "$CASE_NUM" "$FAILED" exit $((FAILED > 0 ? 1 : 0)) diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh b/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh index d8d7303943..3f34a4112a 100755 --- a/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh +++ b/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh @@ -10,7 +10,8 @@ # (scripts/dispatch.sh in the plugin root); this script owns `--registry` and # the duplication tunables it exports for the collector adapters # (CODE_METRICS_DUP_MIN_TOKENS, CODE_METRICS_DUP_MIN_LINES, -# CODE_METRICS_DUP_IGNORE, from `duplication.*` in the resolved config). +# CODE_METRICS_DUP_IGNORE, CODE_METRICS_DUP_MAX_LINES, CODE_METRICS_DUP_MAX_SIZE, +# from `duplication.*` in the resolved config; a null or 0 cap exports empty). # Registries come from every `--registry` plus `duplication.registries`, each # resolved against the repository root; a named registry that does not exist is # a usage error. Exit codes are the dispatcher's: 0 report produced, 2 usage @@ -81,7 +82,8 @@ if [[ -z "$CONFIG" ]]; then --home "${CODE_METRICS_HOME:-${HOME:-/}}" >"$CONFIG" || exit 2 fi -# Three tunables and then one line per configured registry, in that order. +# Five tunables and then one line per configured registry, in that order. A +# cap of null or 0 is exported empty, which the adapter reads as "no cap". mapfile -t DUP < <("${PY[@]}" -c ' import json, sys @@ -93,20 +95,34 @@ def number(key, fallback): return value if isinstance(value, int) and not isinstance(value, bool) else fallback +def cap(key): + value = section.get(key) + if value is None or isinstance(value, bool): + return "" + if isinstance(value, (int, float)): + return str(int(value)) if value > 0 else "" + text = str(value).strip() + return "" if text in ("", "0") else text + + print(number("min_tokens", 50)) print(number("min_lines", 5)) ignore = section.get("ignore") print(",".join(str(item) for item in ignore) if isinstance(ignore, list) else "") +print(cap("max_lines")) +print(cap("max_size")) for registry in section.get("registries") or []: print(str(registry)) ' "$CONFIG") -if [[ ${#DUP[@]} -lt 3 ]]; then +if [[ ${#DUP[@]} -lt 5 ]]; then echo "audit-duplication.sh: the resolved configuration could not be read" >&2 exit 2 fi export CODE_METRICS_DUP_MIN_TOKENS="${DUP[0]}" export CODE_METRICS_DUP_MIN_LINES="${DUP[1]}" export CODE_METRICS_DUP_IGNORE="${DUP[2]}" +export CODE_METRICS_DUP_MAX_LINES="${DUP[3]}" +export CODE_METRICS_DUP_MAX_SIZE="${DUP[4]}" FILTER_ARGS=(--root "$ROOT") resolve_registry() { @@ -119,7 +135,7 @@ resolve_registry() { return 1 fi } -for registry in "${REGISTRY_ARGS[@]:-}" "${DUP[@]:3}"; do +for registry in "${REGISTRY_ARGS[@]:-}" "${DUP[@]:5}"; do [[ -n "$registry" ]] || continue if ! resolved="$(resolve_registry "$registry")"; then echo "audit-duplication.sh: registry not found: $registry" >&2 diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh b/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh index 0490a43f21..cc12fbd8c1 100755 --- a/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh +++ b/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh @@ -62,7 +62,7 @@ trap 'rm -rf "$STUBS" "$EMPTY_PATH" "$WORK"' EXIT cat >"$STUBS/jscpd" <<'STUB' #!/usr/bin/env bash if [[ "${1:-}" == "--version" ]]; then - printf 'jscpd 5.1.2\n' + printf 'jscpd 5.2.0\n' exit 0 fi [[ -z "${CM_TEST_ARGV_LOG:-}" ]] || printf '%s\n' "$*" >>"$CM_TEST_ARGV_LOG" @@ -147,13 +147,15 @@ assert_eq "a missing --registry exits 2" 2 "$?" # 6. The configured tunables reach the collector's command line. "$PY" "$PLUGIN_ROOT/scripts/resolve-config.py" --ladder "$PLUGIN_ROOT/scripts/collector-ladder.tsv" --home "$WORK" >"$WORK/base.json" 2>/dev/null -"$PY" -c 'import json,sys; d=json.load(open(sys.argv[1])); d["duplication"]["min_tokens"] = 77; d["duplication"]["min_lines"] = 9; d["duplication"]["ignore"] = ["**/vendor/**"]; print(json.dumps(d))' "$WORK/base.json" >"$WORK/tuned.json" +"$PY" -c 'import json,sys; d=json.load(open(sys.argv[1])); d["duplication"]["min_tokens"] = 77; d["duplication"]["min_lines"] = 9; d["duplication"]["ignore"] = ["**/vendor/**"]; d["duplication"]["max_size"] = "8kb"; d["duplication"]["max_lines"] = 0; print(json.dumps(d))' "$WORK/base.json" >"$WORK/tuned.json" CM_TEST_ARGV_LOG="$WORK/argv.log" PATH="$STUBS:$EMPTY_PATH" bash "$SCRIPT" --json --all "$CLUSTER" --config "$WORK/tuned.json" >/dev/null 2>&1 assert_eq "the tuned run exits 0" 0 "$?" argv="$(cat "$WORK/argv.log" 2>/dev/null)" assert_contains "min_tokens reaches the collector" "$argv" "--min-tokens 77" assert_contains "min_lines reaches the collector" "$argv" "--min-lines 9" assert_contains "the ignore globs reach the collector" "$argv" "--ignore **/vendor/**" +assert_contains "max_size reaches the collector one byte above the bound" "$argv" "--max-size 8193" +assert_contains "a max_lines of 0 means no cap and reaches the collector as the explicit large value" "$argv" "--max-lines 1000000" # 7. --help prints the usage without running anything. bash "$SCRIPT" --help 2>&1 | grep -q 'audit-duplication.sh \[--json\]' diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py b/plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py index fe5993b4c7..94c9c982d8 100755 --- a/plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py +++ b/plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py @@ -132,8 +132,9 @@ def floor_summary(document: dict[str, Any]) -> dict[str, Any]: that measured nothing keeps its "Measured nothing" headline instead of an unearned zero. """ + # A `partial` row measured what it did not skip, so its zero is earned too. measured = any( - row.get("measure") == "duplication" and row.get("status") == "ok" + row.get("measure") == "duplication" and row.get("status") in ("ok", "partial") for row in document.get("run") or [] ) if measured: diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py b/plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py index f143a6f39f..4cc5d2ea4b 100755 --- a/plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py +++ b/plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py @@ -156,6 +156,22 @@ def test_the_zero_floor_states_zero_once_a_collector_ran(self) -> None: summary = json.loads(result.stdout)["summary"] self.assertEqual((summary["duplicated_lines"], summary["clone_groups"]), (0, 0)) + def test_the_zero_floor_counts_a_partial_lane_as_measured(self) -> None: + doc = document() + doc["run"] = [ + { + "lane": "bash", + "measure": "duplication", + "status": "partial", + "reason": "1 of 3 files skipped by duplication.max_size 1mb / max_lines none", + } + ] + doc["summary"] = {"files": 0, "functions": 0, "over_reference": {}} + result = run(doc, "--root", ".", "--zero-floor") + self.assertEqual(result.returncode, 0, result.stderr) + summary = json.loads(result.stdout)["summary"] + self.assertEqual((summary["duplicated_lines"], summary["clone_groups"]), (0, 0)) + def test_the_zero_floor_leaves_a_run_that_measured_nothing_alone(self) -> None: doc = document() doc["run"] = [ diff --git a/plugins/code-metrics/skills/setup/templates/config-template.yaml b/plugins/code-metrics/skills/setup/templates/config-template.yaml index c72fa99fdc..f3e94b1468 100644 --- a/plugins/code-metrics/skills/setup/templates/config-template.yaml +++ b/plugins/code-metrics/skills/setup/templates/config-template.yaml @@ -28,6 +28,9 @@ duplication: min_lines: 5 ignore: [] registries: [] # sanctioned-replication registries, one relative path per line + max_lines: null # files with more lines are skipped and reported; null (or 0) means no line cap + max_size: 1mb # files larger than this are skipped and reported; jscpd 5's own guard, SonarJS uses 1000kb + rollup_depth: 2 # per-directory rollup rows listed to this depth in the markdown report coverage: artifacts: [] # explicit artifact paths; empty means auto-discover From 37181feb81f7de3d66037af1d41f730d87bed99e Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 16:09:51 +0000 Subject: [PATCH 06/15] feat(code-metrics): merge duplication pairs into clone classes Phase 2 of the duplication audit plan. jscpd (both majors) and PMD CPD report a clone as a pair, so N copies of one fragment arrive as N-1 rows that all name the first copy, and the summary counted the fragment's lines N-1 times. cluster-clones.py runs between the dispatcher and the registry filter and joins pair rows that share an instance with an identical (file, start_line, end_line) and equal lines, keeping the first pair's values, the union of instances sorted by path, and a `clustered` label. Overlap without an identical range stays separate, so a class is never widened past what the detector said was identical. Fixtures under scripts/fixtures/clone-classes (three byte-identical copies; two full copies plus one partial copy) with captures from a real jscpd 5.2.0 run rewritten to repo-relative names. jscpd emits a star for N copies and names the first copy with the same range in every pair, so the plan's offset example is corrected to the partial-copy shape. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01QShQwS5yDYYToBmofdEY5N --- .../code-metrics-duplication-audit/PLAN.md | 14 +- .../aligned/a/shared/shared-utils.sh | 41 ++++ .../aligned/b/shared/shared-utils.sh | 41 ++++ .../aligned/c/shared/shared-utils.sh | 41 ++++ .../fixtures/clone-classes/offset/c1.sh | 38 ++++ .../fixtures/clone-classes/offset/c2.sh | 39 ++++ .../fixtures/clone-classes/offset/c3.sh | 39 ++++ .../fixtures/tool-output/jscpd-aligned3.json | 109 ++++++++++ .../fixtures/tool-output/jscpd-offset3.json | 109 ++++++++++ .../scripts/audit-duplication.sh | 16 +- .../scripts/audit-duplication.test.sh | 34 ++- .../scripts/cluster-clones.py | 149 ++++++++++++++ .../scripts/test_cluster_clones.py | 194 ++++++++++++++++++ 13 files changed, 850 insertions(+), 14 deletions(-) create mode 100644 plugins/code-metrics/scripts/fixtures/clone-classes/aligned/a/shared/shared-utils.sh create mode 100644 plugins/code-metrics/scripts/fixtures/clone-classes/aligned/b/shared/shared-utils.sh create mode 100644 plugins/code-metrics/scripts/fixtures/clone-classes/aligned/c/shared/shared-utils.sh create mode 100644 plugins/code-metrics/scripts/fixtures/clone-classes/offset/c1.sh create mode 100644 plugins/code-metrics/scripts/fixtures/clone-classes/offset/c2.sh create mode 100644 plugins/code-metrics/scripts/fixtures/clone-classes/offset/c3.sh create mode 100644 plugins/code-metrics/scripts/fixtures/tool-output/jscpd-aligned3.json create mode 100644 plugins/code-metrics/scripts/fixtures/tool-output/jscpd-offset3.json create mode 100644 plugins/code-metrics/skills/audit-duplication/scripts/cluster-clones.py create mode 100644 plugins/code-metrics/skills/audit-duplication/scripts/test_cluster_clones.py diff --git a/docs/topics/code-metrics-duplication-audit/PLAN.md b/docs/topics/code-metrics-duplication-audit/PLAN.md index 764e6ae010..2b7de94670 100644 --- a/docs/topics/code-metrics-duplication-audit/PLAN.md +++ b/docs/topics/code-metrics-duplication-audit/PLAN.md @@ -81,8 +81,10 @@ this repository the audit reads clean apart from genuine duplication. `lib/hook-utils.sh -> plugins/*/hooks/hook-utils.sh` that group appears once under `excluded[]` with `duplicated_lines` counted once. - Two groups whose instances overlap without identical line ranges (the `hook-telemetry-sink.sh` - shape, and three copies of one fragment embedded at different offsets) stay separate groups after - the merge. + shape, and a third copy that carries only part of a fragment two full copies share, so the + detector names the first copy with two different ranges) stay separate groups after the merge. + Three full copies at different line offsets are one class: jscpd pairs each later copy with the + first and names that first copy with the same range in every pair (verified against 5.2.0). - A cluster line excludes a group only when every instance's root-relative path matches the canonical path or one of the members (literal or glob) and the instances' directories are all distinct; two copies inside one directory still count as duplication; a single-token line behaves @@ -298,7 +300,7 @@ Review: code-design shows `bash` `ok` and `typescript` `partial`, and the typescript reason names `plugins/miro/dist/index.min.js`; the same under 4.3.0. -### Phase 2: Merge pairs into clone classes [TODO] +### Phase 2: Merge pairs into clone classes [DONE] Review: code-design @@ -312,8 +314,10 @@ Review: code-design 2. `audit-duplication.sh`: pipe `report.json` through `cluster-clones.py` before `registry-filter.py`. 3. Fixtures, outside `fixtures/sources` so no suite that scopes that tree changes its counts: `scripts/fixtures/clone-classes/aligned/{a,b,c}/shared/shared-utils.sh` (three byte-identical - copies) and `scripts/fixtures/clone-classes/offset/{c1,c2,c3}.sh` (one 41-line fragment at - offsets 1, 2, 3 with different flanking lines); committed captures + copies) and `scripts/fixtures/clone-classes/offset/{c1,c2,c3}.sh` (`c1` and `c2` carry one 34-line + fragment at different line offsets with different flanking lines; `c3` carries only the first + 24 lines of it, so its pair names `c1` with a shorter range and the two pairs do not merge); + committed captures `scripts/fixtures/tool-output/jscpd-aligned3.json` and `jscpd-offset3.json` produced by a real jscpd 5.2.0 run, then rewritten to repo-relative names (the adapter passes `--absolute`, so the raw capture carries machine paths, and the stub replays the file regardless of input). diff --git a/plugins/code-metrics/scripts/fixtures/clone-classes/aligned/a/shared/shared-utils.sh b/plugins/code-metrics/scripts/fixtures/clone-classes/aligned/a/shared/shared-utils.sh new file mode 100644 index 0000000000..027c163dad --- /dev/null +++ b/plugins/code-metrics/scripts/fixtures/clone-classes/aligned/a/shared/shared-utils.sh @@ -0,0 +1,41 @@ +# shellcheck shell=bash +# Fixture source for the code-metrics duplication suites: a helper vendored +# byte-identical into two sibling plugin directories, standing in for a +# repository that deliberately replicates one path across its plugins. Never +# executed, so it carries no shebang and no exec bit; kept lint-clean on +# purpose. The copy under the sibling directory is byte-for-byte this file. + +HARVEST_LABEL="harvest" + +announce_start() { + local subject="$1" + printf 'start %s %s\n' "$HARVEST_LABEL" "$subject" +} + +announce_finish() { + local subject="$1" + local outcome="${2:-unknown}" + printf 'finish %s %s %s\n' "$HARVEST_LABEL" "$subject" "$outcome" +} + +collect_orchard() { + local basket="$1" + shift + local apple + for apple in "$@"; do + if [[ -z "$apple" ]]; then + continue + fi + printf '%s/%s\n' "$basket" "$apple" + done +} + +measure_basket() { + local basket="$1" + if [[ -d "$basket" ]]; then + find "$basket" -type f | wc -l + return 0 + fi + printf '0\n' + return 1 +} diff --git a/plugins/code-metrics/scripts/fixtures/clone-classes/aligned/b/shared/shared-utils.sh b/plugins/code-metrics/scripts/fixtures/clone-classes/aligned/b/shared/shared-utils.sh new file mode 100644 index 0000000000..027c163dad --- /dev/null +++ b/plugins/code-metrics/scripts/fixtures/clone-classes/aligned/b/shared/shared-utils.sh @@ -0,0 +1,41 @@ +# shellcheck shell=bash +# Fixture source for the code-metrics duplication suites: a helper vendored +# byte-identical into two sibling plugin directories, standing in for a +# repository that deliberately replicates one path across its plugins. Never +# executed, so it carries no shebang and no exec bit; kept lint-clean on +# purpose. The copy under the sibling directory is byte-for-byte this file. + +HARVEST_LABEL="harvest" + +announce_start() { + local subject="$1" + printf 'start %s %s\n' "$HARVEST_LABEL" "$subject" +} + +announce_finish() { + local subject="$1" + local outcome="${2:-unknown}" + printf 'finish %s %s %s\n' "$HARVEST_LABEL" "$subject" "$outcome" +} + +collect_orchard() { + local basket="$1" + shift + local apple + for apple in "$@"; do + if [[ -z "$apple" ]]; then + continue + fi + printf '%s/%s\n' "$basket" "$apple" + done +} + +measure_basket() { + local basket="$1" + if [[ -d "$basket" ]]; then + find "$basket" -type f | wc -l + return 0 + fi + printf '0\n' + return 1 +} diff --git a/plugins/code-metrics/scripts/fixtures/clone-classes/aligned/c/shared/shared-utils.sh b/plugins/code-metrics/scripts/fixtures/clone-classes/aligned/c/shared/shared-utils.sh new file mode 100644 index 0000000000..027c163dad --- /dev/null +++ b/plugins/code-metrics/scripts/fixtures/clone-classes/aligned/c/shared/shared-utils.sh @@ -0,0 +1,41 @@ +# shellcheck shell=bash +# Fixture source for the code-metrics duplication suites: a helper vendored +# byte-identical into two sibling plugin directories, standing in for a +# repository that deliberately replicates one path across its plugins. Never +# executed, so it carries no shebang and no exec bit; kept lint-clean on +# purpose. The copy under the sibling directory is byte-for-byte this file. + +HARVEST_LABEL="harvest" + +announce_start() { + local subject="$1" + printf 'start %s %s\n' "$HARVEST_LABEL" "$subject" +} + +announce_finish() { + local subject="$1" + local outcome="${2:-unknown}" + printf 'finish %s %s %s\n' "$HARVEST_LABEL" "$subject" "$outcome" +} + +collect_orchard() { + local basket="$1" + shift + local apple + for apple in "$@"; do + if [[ -z "$apple" ]]; then + continue + fi + printf '%s/%s\n' "$basket" "$apple" + done +} + +measure_basket() { + local basket="$1" + if [[ -d "$basket" ]]; then + find "$basket" -type f | wc -l + return 0 + fi + printf '0\n' + return 1 +} diff --git a/plugins/code-metrics/scripts/fixtures/clone-classes/offset/c1.sh b/plugins/code-metrics/scripts/fixtures/clone-classes/offset/c1.sh new file mode 100644 index 0000000000..27b4041c3d --- /dev/null +++ b/plugins/code-metrics/scripts/fixtures/clone-classes/offset/c1.sh @@ -0,0 +1,38 @@ +# shellcheck shell=bash +# Offset fixture: the shared fragment begins on line 3 of this copy. +HARVEST_LABEL="harvest" + +announce_start() { + local subject="$1" + printf 'start %s %s\n' "$HARVEST_LABEL" "$subject" +} + +announce_finish() { + local subject="$1" + local outcome="${2:-unknown}" + printf 'finish %s %s %s\n' "$HARVEST_LABEL" "$subject" "$outcome" +} + +collect_orchard() { + local basket="$1" + shift + local apple + for apple in "$@"; do + if [[ -z "$apple" ]]; then + continue + fi + printf '%s/%s\n' "$basket" "$apple" + done +} + +measure_basket() { + local basket="$1" + if [[ -d "$basket" ]]; then + find "$basket" -type f | wc -l + return 0 + fi + printf '0\n' + return 1 +} + +announce_start "offset-one" diff --git a/plugins/code-metrics/scripts/fixtures/clone-classes/offset/c2.sh b/plugins/code-metrics/scripts/fixtures/clone-classes/offset/c2.sh new file mode 100644 index 0000000000..176809c895 --- /dev/null +++ b/plugins/code-metrics/scripts/fixtures/clone-classes/offset/c2.sh @@ -0,0 +1,39 @@ +# shellcheck shell=bash +# Offset fixture: the shared fragment begins on line 4 of this copy, one +# line lower than in the first copy, so no instance range aligns with it. +HARVEST_LABEL="harvest" + +announce_start() { + local subject="$1" + printf 'start %s %s\n' "$HARVEST_LABEL" "$subject" +} + +announce_finish() { + local subject="$1" + local outcome="${2:-unknown}" + printf 'finish %s %s %s\n' "$HARVEST_LABEL" "$subject" "$outcome" +} + +collect_orchard() { + local basket="$1" + shift + local apple + for apple in "$@"; do + if [[ -z "$apple" ]]; then + continue + fi + printf '%s/%s\n' "$basket" "$apple" + done +} + +measure_basket() { + local basket="$1" + if [[ -d "$basket" ]]; then + find "$basket" -type f | wc -l + return 0 + fi + printf '0\n' + return 1 +} + +announce_finish "offset-two" "done" diff --git a/plugins/code-metrics/scripts/fixtures/clone-classes/offset/c3.sh b/plugins/code-metrics/scripts/fixtures/clone-classes/offset/c3.sh new file mode 100644 index 0000000000..f4ff674ba4 --- /dev/null +++ b/plugins/code-metrics/scripts/fixtures/clone-classes/offset/c3.sh @@ -0,0 +1,39 @@ +# shellcheck shell=bash +# Offset fixture: this copy carries only the first part of the shared +# fragment, beginning on line 5, so the range it shares with the first copy +# is shorter than the range the first and second copies share. The two pairs +# jscpd reports therefore name the first copy with two different ranges and +# stay two clone groups: the merge joins pairs on an identical instance, not +# on an overlapping one. +HARVEST_LABEL="harvest" + +announce_start() { + local subject="$1" + printf 'start %s %s\n' "$HARVEST_LABEL" "$subject" +} + +announce_finish() { + local subject="$1" + local outcome="${2:-unknown}" + printf 'finish %s %s %s\n' "$HARVEST_LABEL" "$subject" "$outcome" +} + +collect_orchard() { + local basket="$1" + shift + local apple + for apple in "$@"; do + if [[ -z "$apple" ]]; then + continue + fi + printf '%s/%s\n' "$basket" "$apple" + done +} + +weigh_basket() { + local basket="$1" + local weight="${2:-0}" + printf 'weigh %s %s\n' "$basket" "$weight" +} + +weigh_basket "offset-three" 7 diff --git a/plugins/code-metrics/scripts/fixtures/tool-output/jscpd-aligned3.json b/plugins/code-metrics/scripts/fixtures/tool-output/jscpd-aligned3.json new file mode 100644 index 0000000000..030fd71e1b --- /dev/null +++ b/plugins/code-metrics/scripts/fixtures/tool-output/jscpd-aligned3.json @@ -0,0 +1,109 @@ +{ + "duplicates": [ + { + "firstFile": { + "end": 41, + "endLoc": { + "column": 1, + "line": 41, + "position": 1002 + }, + "name": "plugins/code-metrics/scripts/fixtures/clone-classes/aligned/a/shared/shared-utils.sh", + "start": 1, + "startLoc": { + "column": 0, + "line": 1, + "position": 0 + } + }, + "format": "bash", + "fragment": "# shellcheck shell=bash\n# Fixture source for the code-metrics duplication suites: a helper vendored\n# byte-identical into two sibling plugin directories, standing in for a\n# repository that deliberately replicates one path across its plugins. Never\n# executed, so it carries no shebang and no exec bit; kept lint-clean on\n# purpose. The copy under the sibling directory is byte-for-byte this file.\n\nHARVEST_LABEL=\"harvest\"\n\nannounce_start() {\n local subject=\"$1\"\n printf 'start %s %s\\n' \"$HARVEST_LABEL\" \"$subject\"\n}\n\nannounce_finish() {\n local subject=\"$1\"\n local outcome=\"${2:-unknown}\"\n printf 'finish %s %s %s\\n' \"$HARVEST_LABEL\" \"$subject\" \"$outcome\"\n}\n\ncollect_orchard() {\n local basket=\"$1\"\n shift\n local apple\n for apple in \"$@\"; do\n if [[ -z \"$apple\" ]]; then\n continue\n fi\n printf '%s/%s\\n' \"$basket\" \"$apple\"\n done\n}\n\nmeasure_basket() {\n local basket=\"$1\"\n if [[ -d \"$basket\" ]]; then\n find \"$basket\" -type f | wc -l\n return 0\n fi\n printf '0\\n'\n return 1\n}", + "isNew": false, + "kind": "exact", + "lines": 41, + "secondFile": { + "end": 41, + "endLoc": { + "column": 1, + "line": 41, + "position": 1002 + }, + "name": "plugins/code-metrics/scripts/fixtures/clone-classes/aligned/b/shared/shared-utils.sh", + "start": 1, + "startLoc": { + "column": 0, + "line": 1, + "position": 0 + } + }, + "tokens": 110 + }, + { + "firstFile": { + "end": 41, + "endLoc": { + "column": 1, + "line": 41, + "position": 1002 + }, + "name": "plugins/code-metrics/scripts/fixtures/clone-classes/aligned/a/shared/shared-utils.sh", + "start": 1, + "startLoc": { + "column": 0, + "line": 1, + "position": 0 + } + }, + "format": "bash", + "fragment": "# shellcheck shell=bash\n# Fixture source for the code-metrics duplication suites: a helper vendored\n# byte-identical into two sibling plugin directories, standing in for a\n# repository that deliberately replicates one path across its plugins. Never\n# executed, so it carries no shebang and no exec bit; kept lint-clean on\n# purpose. The copy under the sibling directory is byte-for-byte this file.\n\nHARVEST_LABEL=\"harvest\"\n\nannounce_start() {\n local subject=\"$1\"\n printf 'start %s %s\\n' \"$HARVEST_LABEL\" \"$subject\"\n}\n\nannounce_finish() {\n local subject=\"$1\"\n local outcome=\"${2:-unknown}\"\n printf 'finish %s %s %s\\n' \"$HARVEST_LABEL\" \"$subject\" \"$outcome\"\n}\n\ncollect_orchard() {\n local basket=\"$1\"\n shift\n local apple\n for apple in \"$@\"; do\n if [[ -z \"$apple\" ]]; then\n continue\n fi\n printf '%s/%s\\n' \"$basket\" \"$apple\"\n done\n}\n\nmeasure_basket() {\n local basket=\"$1\"\n if [[ -d \"$basket\" ]]; then\n find \"$basket\" -type f | wc -l\n return 0\n fi\n printf '0\\n'\n return 1\n}", + "isNew": false, + "kind": "exact", + "lines": 41, + "secondFile": { + "end": 41, + "endLoc": { + "column": 1, + "line": 41, + "position": 1002 + }, + "name": "plugins/code-metrics/scripts/fixtures/clone-classes/aligned/c/shared/shared-utils.sh", + "start": 1, + "startLoc": { + "column": 0, + "line": 1, + "position": 0 + } + }, + "tokens": 110 + } + ], + "statistics": { + "detectionDate": "2026-09-11T16:00:18.373Z", + "formats": { + "bash": { + "clones": 2, + "duplicatedLines": 80, + "duplicatedTokens": 220, + "lines": 123, + "newClones": 0, + "newDuplicatedLines": 0, + "percentage": 65.04065040650406, + "percentageTokens": 66.66666666666666, + "sources": 3, + "tokens": 330 + } + }, + "total": { + "clones": 2, + "duplicatedLines": 80, + "duplicatedTokens": 220, + "lines": 123, + "newClones": 0, + "newDuplicatedLines": 0, + "percentage": 65.04065040650406, + "percentageTokens": 66.66666666666666, + "sources": 3, + "tokens": 330 + } + } +} \ No newline at end of file diff --git a/plugins/code-metrics/scripts/fixtures/tool-output/jscpd-offset3.json b/plugins/code-metrics/scripts/fixtures/tool-output/jscpd-offset3.json new file mode 100644 index 0000000000..051ed487e1 --- /dev/null +++ b/plugins/code-metrics/scripts/fixtures/tool-output/jscpd-offset3.json @@ -0,0 +1,109 @@ +{ + "duplicates": [ + { + "firstFile": { + "end": 36, + "endLoc": { + "column": 1, + "line": 36, + "position": 696 + }, + "name": "plugins/code-metrics/scripts/fixtures/clone-classes/offset/c1.sh", + "start": 3, + "startLoc": { + "column": 0, + "line": 3, + "position": 93 + } + }, + "format": "bash", + "fragment": "HARVEST_LABEL=\"harvest\"\n\nannounce_start() {\n local subject=\"$1\"\n printf 'start %s %s\\n' \"$HARVEST_LABEL\" \"$subject\"\n}\n\nannounce_finish() {\n local subject=\"$1\"\n local outcome=\"${2:-unknown}\"\n printf 'finish %s %s %s\\n' \"$HARVEST_LABEL\" \"$subject\" \"$outcome\"\n}\n\ncollect_orchard() {\n local basket=\"$1\"\n shift\n local apple\n for apple in \"$@\"; do\n if [[ -z \"$apple\" ]]; then\n continue\n fi\n printf '%s/%s\\n' \"$basket\" \"$apple\"\n done\n}\n\nmeasure_basket() {\n local basket=\"$1\"\n if [[ -d \"$basket\" ]]; then\n find \"$basket\" -type f | wc -l\n return 0\n fi\n printf '0\\n'\n return 1\n}", + "isNew": false, + "kind": "exact", + "lines": 34, + "secondFile": { + "end": 37, + "endLoc": { + "column": 1, + "line": 37, + "position": 774 + }, + "name": "plugins/code-metrics/scripts/fixtures/clone-classes/offset/c2.sh", + "start": 4, + "startLoc": { + "column": 0, + "line": 4, + "position": 171 + } + }, + "tokens": 104 + }, + { + "firstFile": { + "end": 26, + "endLoc": { + "column": 1, + "line": 26, + "position": 545 + }, + "name": "plugins/code-metrics/scripts/fixtures/clone-classes/offset/c1.sh", + "start": 3, + "startLoc": { + "column": 0, + "line": 3, + "position": 93 + } + }, + "format": "bash", + "fragment": "HARVEST_LABEL=\"harvest\"\n\nannounce_start() {\n local subject=\"$1\"\n printf 'start %s %s\\n' \"$HARVEST_LABEL\" \"$subject\"\n}\n\nannounce_finish() {\n local subject=\"$1\"\n local outcome=\"${2:-unknown}\"\n printf 'finish %s %s %s\\n' \"$HARVEST_LABEL\" \"$subject\" \"$outcome\"\n}\n\ncollect_orchard() {\n local basket=\"$1\"\n shift\n local apple\n for apple in \"$@\"; do\n if [[ -z \"$apple\" ]]; then\n continue\n fi\n printf '%s/%s\\n' \"$basket\" \"$apple\"\n done\n}", + "isNew": false, + "kind": "exact", + "lines": 24, + "secondFile": { + "end": 31, + "endLoc": { + "column": 1, + "line": 31, + "position": 877 + }, + "name": "plugins/code-metrics/scripts/fixtures/clone-classes/offset/c3.sh", + "start": 8, + "startLoc": { + "column": 0, + "line": 8, + "position": 425 + } + }, + "tokens": 69 + } + ], + "statistics": { + "detectionDate": "2026-09-11T16:02:53.170Z", + "formats": { + "bash": { + "clones": 2, + "duplicatedLines": 56, + "duplicatedTokens": 173, + "lines": 116, + "newClones": 0, + "newDuplicatedLines": 0, + "percentage": 48.275862068965516, + "percentageTokens": 55.095541401273884, + "sources": 3, + "tokens": 314 + } + }, + "total": { + "clones": 2, + "duplicatedLines": 56, + "duplicatedTokens": 173, + "lines": 116, + "newClones": 0, + "newDuplicatedLines": 0, + "percentage": 48.275862068965516, + "percentageTokens": 55.095541401273884, + "sources": 3, + "tokens": 314 + } + } +} \ No newline at end of file diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh b/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh index 3f34a4112a..6753f11343 100755 --- a/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh +++ b/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh @@ -7,8 +7,9 @@ # # Prints the markdown report; `--json` prints the `code-metrics/v1` document # instead. Scope, lanes, and the collector ladder are the dispatcher's -# (scripts/dispatch.sh in the plugin root); this script owns `--registry` and -# the duplication tunables it exports for the collector adapters +# (scripts/dispatch.sh in the plugin root); this script owns the merge of the +# detector's clone pairs into clone classes (cluster-clones.py), `--registry`, +# and the duplication tunables it exports for the collector adapters # (CODE_METRICS_DUP_MIN_TOKENS, CODE_METRICS_DUP_MIN_LINES, # CODE_METRICS_DUP_IGNORE, CODE_METRICS_DUP_MAX_LINES, CODE_METRICS_DUP_MAX_SIZE, # from `duplication.*` in the resolved config; a null or 0 cap exports empty). @@ -22,6 +23,7 @@ SCRIPT_DIR="$(cd "${BASH_SOURCE[0]%/*}" && pwd)" PLUGIN_ROOT="${CLAUDE_PLUGIN_ROOT:-$(cd "$SCRIPT_DIR/../../.." && pwd)}" DISPATCH="$PLUGIN_ROOT/scripts/dispatch.sh" REPORT="$PLUGIN_ROOT/scripts/report.py" +CLUSTER="$SCRIPT_DIR/cluster-clones.py" FILTER="$SCRIPT_DIR/registry-filter.py" JSON=0 @@ -51,7 +53,7 @@ while [[ $# -gt 0 ]]; do shift 2 ;; --help | -h) - sed -n '2,17p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' >&2 + sed -n '2,19p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' >&2 exit 0 ;; *) @@ -148,9 +150,11 @@ bash "$DISPATCH" audit-duplication --measures duplication --config "$CONFIG" ${P rc=$? [[ $rc -eq 0 || $rc -eq 3 ]] || exit "$rc" -# Exclude the declared replication, recompute the totals from what survived, -# then state the zero the recomputation drops when every group was excluded. -"${PY[@]}" "$FILTER" "${FILTER_ARGS[@]}" <"$WORK/report.json" >"$WORK/filtered.json" || exit 2 +# Merge the pairs the detector reports into clone classes, exclude the declared +# replication, recompute the totals from what survived, then state the zero the +# recomputation drops when every group was excluded. +"${PY[@]}" "$CLUSTER" <"$WORK/report.json" >"$WORK/clustered.json" || exit 2 +"${PY[@]}" "$FILTER" "${FILTER_ARGS[@]}" <"$WORK/clustered.json" >"$WORK/filtered.json" || exit 2 "${PY[@]}" "$REPORT" resummarize <"$WORK/filtered.json" >"$WORK/summed.json" || exit 2 "${PY[@]}" "$FILTER" --zero-floor --root "$ROOT" <"$WORK/summed.json" >"$WORK/final.json" || exit 2 diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh b/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh index cc12fbd8c1..48b0ee5760 100755 --- a/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh +++ b/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh @@ -9,7 +9,11 @@ # adapter passes and exits 1, the reporting exit code the contract says is not # a failure (design T13; no executable is committed). The fixture cluster is # scripts/fixtures/sources/cluster/{alpha,beta}/shared/shared-utils.sh and the -# registry that sanctions it is scripts/fixtures/registry/cluster.txt. +# registry that sanctions it is scripts/fixtures/registry/cluster.txt. The +# three-copy cases swap in the captures jscpd-aligned3.json and +# jscpd-offset3.json, real jscpd 5.2.0 runs over +# scripts/fixtures/clone-classes/{aligned,offset} rewritten to repo-relative +# names. # # The last case is the Brief's own: this repository's real # plugins/*/hooks/hook-utils.sh cluster against @@ -157,11 +161,35 @@ assert_contains "the ignore globs reach the collector" "$argv" "--ignore **/vend assert_contains "max_size reaches the collector one byte above the bound" "$argv" "--max-size 8193" assert_contains "a max_lines of 0 means no cap and reaches the collector as the explicit large value" "$argv" "--max-lines 1000000" -# 7. --help prints the usage without running anything. +# 7. Three byte-identical copies are one clone class, its lines counted once. +# jscpd pairs each later copy with the first, so the capture holds two pairs +# that name the same instance of copy `a`. +ALIGNED="$FIXTURES/clone-classes/aligned" +out="$(CM_TEST_CAPTURE="$REPO_ROOT/$FIXTURES/tool-output/jscpd-aligned3.json" PATH="$STUBS:$EMPTY_PATH" bash "$SCRIPT" --json --all "$ALIGNED")" +assert_eq "the aligned three-copy run exits 0" 0 "$?" +if printf '%s' "$out" | "$PY" -c 'import json,sys; d=json.load(sys.stdin); assert d["summary"]["clone_groups"] == 1 and d["summary"]["duplicated_lines"] == 41, d["summary"]; row = d["measures"][0]; assert len(row["instances"]) == 3 and "clustered" in row["labels"], row' 2>/dev/null; then + pass "three aligned copies are one clone class with the lines counted once" +else + fail "three aligned copies are one clone class with the lines counted once" "clone_groups 1, duplicated_lines 41, three instances" "$(printf '%s' "$out" | head -c 600)" +fi + +# 8. A third copy that shares only part of the fragment stays its own group: +# the two pairs name copy `c1` with different ranges, and overlap is not +# identity. +OFFSET="$FIXTURES/clone-classes/offset" +out="$(CM_TEST_CAPTURE="$REPO_ROOT/$FIXTURES/tool-output/jscpd-offset3.json" PATH="$STUBS:$EMPTY_PATH" bash "$SCRIPT" --json --all "$OFFSET")" +assert_eq "the offset three-copy run exits 0" 0 "$?" +if printf '%s' "$out" | "$PY" -c 'import json,sys; d=json.load(sys.stdin); assert d["summary"]["clone_groups"] == 2 and d["summary"]["duplicated_lines"] == 58, d["summary"]; assert all(len(r["instances"]) == 2 and "clustered" not in r["labels"] for r in d["measures"]), d["measures"]' 2>/dev/null; then + pass "a partial third copy stays a second clone group" +else + fail "a partial third copy stays a second clone group" "clone_groups 2, duplicated_lines 58, two instances each" "$(printf '%s' "$out" | head -c 600)" +fi + +# 9. --help prints the usage without running anything. bash "$SCRIPT" --help 2>&1 | grep -q 'audit-duplication.sh \[--json\]' assert_eq "--help prints usage" 0 "$?" -# 8. The Brief's case: this repository's own vendored hook-utils cluster. +# 10. The Brief's case: this repository's own vendored hook-utils cluster. # The jscpd on PATH has to be a working detector, not another suite's stub or # a replaying fake: the probe copies one fixture into two directories under a # name nothing else uses and requires the report to name it back. diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/cluster-clones.py b/plugins/code-metrics/skills/audit-duplication/scripts/cluster-clones.py new file mode 100644 index 0000000000..f860dfa9b8 --- /dev/null +++ b/plugins/code-metrics/skills/audit-duplication/scripts/cluster-clones.py @@ -0,0 +1,149 @@ +#!/usr/bin/env python3 +"""Merge the clone pairs a detector reports into clone classes. + + cluster-clones.py [< report.json] + +Reads a `code-metrics/v1` document on stdin and prints it back with the +two-instance clone-group rows that share an identical instance merged into one +row per clone class. jscpd (both majors) and PMD CPD report a clone as a PAIR, +so N byte-identical copies of one fragment arrive as N-1 rows that all name +the same instance of the first copy, and a summary derived from those rows +would count the fragment's lines N-1 times. A clone class is the union of +every pair that shares a code portion (Roy and Cordy 2007 s.6, citing Rieger, +Ducasse and Lanza 2004; Roy, Cordy and Koschke 2009 aggregate pairs into +classes in post-processing), and that closure is exact for the byte-identical +clones a token detector reports. + +Two rows join when they share an instance with identical `(file, start_line, +end_line)` and equal `values.lines`. Overlap is not enough: a pair whose +shared file is named with a different range (a third copy that carries only +part of the fragment) stays its own group, so a class is never widened past +what the detector said was identical. The merged row keeps the first row's +`values`, so the fragment's lines count once, carries the union of the +instances sorted by `(file, start_line)`, and appends `clustered` to `labels`. +A row with three or more instances is already a class and passes through, as +does every row without `instances`, and every row keeps its position. Rows +join whatever their `collector`, so a pair another detector reported merges +too. `summary` is left alone: the caller recomputes it with `report.py +resummarize`. + +Exit 0 when the document was printed, 2 when stdin is not a JSON document. +""" + +from __future__ import annotations + +import json +import sys +from typing import Any + +MIN_PYTHON = (3, 9) +LABEL = "clustered" + + +def instance_key(instance: dict[str, Any]) -> tuple[str, Any, Any]: + return ( + str(instance.get("file", "")).replace("\\", "/"), + instance.get("start_line"), + instance.get("end_line"), + ) + + +def _sort_key(instance: dict[str, Any]) -> tuple[str, int]: + start = instance.get("start_line") + return (instance_key(instance)[0], start if isinstance(start, int) else -1) + + +def _is_pair(row: dict[str, Any]) -> bool: + instances = row.get("instances") + return isinstance(instances, list) and len(instances) == 2 + + +def cluster(measures: list[dict[str, Any]]) -> list[dict[str, Any]]: + """Return `measures` with pair rows that share an identical instance merged.""" + parent: dict[int, int] = { + index: index for index, row in enumerate(measures) if _is_pair(row) + } + + def find(index: int) -> int: + while parent[index] != index: + parent[index] = parent[parent[index]] + index = parent[index] + return index + + def union(left: int, right: int) -> None: + left, right = find(left), find(right) + if left != right: + # The lower index stays the root, so a class is emitted where its + # first pair stood and its `values` are that first pair's. + parent[max(left, right)] = min(left, right) + + seen: dict[tuple[Any, ...], int] = {} + for index in parent: + row = measures[index] + lines = (row.get("values") or {}).get("lines") + for instance in row["instances"]: + key = (instance_key(instance), lines) + if key in seen: + union(seen[key], index) + else: + seen[key] = index + + members: dict[int, list[int]] = {} + for index in parent: + members.setdefault(find(index), []).append(index) + + output: list[dict[str, Any]] = [] + for index, row in enumerate(measures): + if index not in parent: + output.append(row) + continue + root = find(index) + if root != index: + continue + group = members[root] + if len(group) == 1: + output.append(row) + continue + instances: dict[tuple[str, Any, Any], dict[str, Any]] = {} + for member in group: + for instance in measures[member]["instances"]: + instances.setdefault(instance_key(instance), instance) + merged = dict(row) + merged["instances"] = sorted(instances.values(), key=_sort_key) + labels = [str(label) for label in row.get("labels") or []] + if LABEL not in labels: + labels.append(LABEL) + merged["labels"] = labels + output.append(merged) + return output + + +def main(argv: list[str]) -> int: + if argv: + print("usage: cluster-clones.py < report.json", file=sys.stderr) + return 2 + try: + document = json.load(sys.stdin) + except (json.JSONDecodeError, ValueError) as exc: + print( + f"cluster-clones.py: stdin is not a JSON document ({exc})", file=sys.stderr + ) + return 2 + if not isinstance(document, dict): + print("cluster-clones.py: stdin is not a JSON object", file=sys.stderr) + return 2 + measures = document.get("measures") + if isinstance(measures, list): + document["measures"] = cluster(measures) + print(json.dumps(document, indent=2)) + return 0 + + +if __name__ == "__main__": + if sys.version_info < MIN_PYTHON: + print( + "cluster-clones.py needs Python %d.%d or later" % MIN_PYTHON, + file=sys.stderr, + ) + sys.exit(2) + sys.exit(main(sys.argv[1:])) diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/test_cluster_clones.py b/plugins/code-metrics/skills/audit-duplication/scripts/test_cluster_clones.py new file mode 100644 index 0000000000..de65921e96 --- /dev/null +++ b/plugins/code-metrics/skills/audit-duplication/scripts/test_cluster_clones.py @@ -0,0 +1,194 @@ +#!/usr/bin/env python3 +"""Output-based tests for cluster-clones.py at its command line. + +The post-pass is a pure function from a document to a document, so every case +drives the script through subprocess with a small document on stdin (design +T13, the one seam per script). The committed captures +scripts/fixtures/tool-output/jscpd-aligned3.json and jscpd-offset3.json cover +the shapes jscpd itself reports for three copies; these cases cover the rule. +""" + +from __future__ import annotations + +import json +import subprocess +import sys +import unittest +from pathlib import Path + +SCRIPT = Path(__file__).resolve().parent / "cluster-clones.py" + + +def instance(path: str, start: int, end: int) -> dict: + return {"file": path, "start_line": start, "end_line": end} + + +def pair( + first: tuple[str, int, int], + second: tuple[str, int, int], + lines: int = 41, + collector: str = "jscpd", + labels: list[str] | None = None, +) -> dict: + return { + "file": None, + "function": None, + "lane": "bash", + "instances": [instance(*first), instance(*second)], + "values": {"lines": lines, "tokens": 110}, + "collector": collector, + "labels": ["token-based"] if labels is None else labels, + } + + +def document(*rows: dict) -> dict: + return { + "schema": "code-metrics/v1", + "skill": "audit-duplication", + "measures": list(rows), + "excluded": [], + "summary": {"files": 0, "functions": 0, "over_reference": {}}, + } + + +def run(doc: dict | str, *args: str) -> subprocess.CompletedProcess: + return subprocess.run( + [sys.executable, str(SCRIPT), *args], + input=doc if isinstance(doc, str) else json.dumps(doc), + capture_output=True, + text=True, + check=False, + ) + + +def measures(result: subprocess.CompletedProcess) -> list[dict]: + return json.loads(result.stdout)["measures"] + + +A = "plugins/x/a/shared/shared-utils.sh" +B = "plugins/x/b/shared/shared-utils.sh" +C = "plugins/x/c/shared/shared-utils.sh" + + +class ClusterClonesTests(unittest.TestCase): + def test_three_aligned_copies_collapse_to_one_class(self) -> None: + # jscpd pairs every later copy with the first, so two rows name the + # same instance of the first copy. + result = run( + document(pair((A, 1, 41), (B, 1, 41)), pair((A, 1, 41), (C, 1, 41))) + ) + self.assertEqual(result.returncode, 0, result.stderr) + rows = measures(result) + self.assertEqual(len(rows), 1) + self.assertEqual([i["file"] for i in rows[0]["instances"]], [A, B, C]) + self.assertEqual(rows[0]["values"], {"lines": 41, "tokens": 110}) + self.assertEqual(rows[0]["labels"], ["token-based", "clustered"]) + + def test_the_lines_of_a_merged_class_count_once(self) -> None: + result = run( + document(pair((A, 1, 41), (B, 1, 41)), pair((A, 1, 41), (C, 1, 41))) + ) + rows = measures(result) + self.assertEqual(sum(row["values"]["lines"] for row in rows), 41) + + def test_overlap_without_an_identical_range_stays_two_groups(self) -> None: + # The third copy carries only part of the fragment, so its pair names + # the first copy with a shorter range. + result = run( + document( + pair(("c1.sh", 3, 36), ("c2.sh", 4, 37), lines=34), + pair(("c1.sh", 3, 26), ("c3.sh", 5, 28), lines=24), + ) + ) + rows = measures(result) + self.assertEqual(len(rows), 2) + self.assertTrue(all(len(row["instances"]) == 2 for row in rows)) + self.assertTrue(all("clustered" not in row["labels"] for row in rows)) + + def test_an_identical_range_with_different_lines_does_not_join(self) -> None: + result = run( + document( + pair((A, 1, 41), (B, 1, 41), lines=41), + pair((A, 1, 41), (C, 1, 41), lines=40), + ) + ) + self.assertEqual(len(measures(result)), 2) + + def test_a_three_instance_row_passes_through_unchanged(self) -> None: + row = pair((A, 1, 41), (B, 1, 41)) + row["instances"].append(instance(C, 1, 41)) + other = pair((A, 1, 41), ("plugins/x/d/shared/shared-utils.sh", 1, 41)) + result = run(document(row, other)) + rows = measures(result) + self.assertEqual(rows[0], row) + self.assertEqual(rows[1], other) + + def test_a_pair_from_another_collector_joins_on_an_identical_instance( + self, + ) -> None: + result = run( + document( + pair((A, 1, 41), (B, 1, 41)), + pair((A, 1, 41), (C, 1, 41), collector="cpd", labels=["cpd"]), + ) + ) + rows = measures(result) + self.assertEqual(len(rows), 1) + self.assertEqual(rows[0]["collector"], "jscpd") + self.assertEqual([i["file"] for i in rows[0]["instances"]], [A, B, C]) + + def test_merged_instances_are_sorted_by_path_then_start_line(self) -> None: + result = run( + document(pair((C, 1, 41), (B, 1, 41)), pair((A, 1, 41), (B, 1, 41))) + ) + rows = measures(result) + self.assertEqual(len(rows), 1) + self.assertEqual([i["file"] for i in rows[0]["instances"]], [A, B, C]) + + def test_a_chain_of_pairs_closes_into_one_class(self) -> None: + result = run( + document(pair((A, 1, 41), (B, 1, 41)), pair((B, 1, 41), (C, 1, 41))) + ) + rows = measures(result) + self.assertEqual(len(rows), 1) + self.assertEqual([i["file"] for i in rows[0]["instances"]], [A, B, C]) + + def test_the_class_is_emitted_where_its_first_pair_stood(self) -> None: + unrelated = pair(("y.sh", 1, 10), ("z.sh", 1, 10), lines=10) + result = run( + document( + pair((A, 1, 41), (B, 1, 41)), unrelated, pair((A, 1, 41), (C, 1, 41)) + ) + ) + rows = measures(result) + self.assertEqual(len(rows), 2) + self.assertEqual(len(rows[0]["instances"]), 3) + self.assertEqual(rows[1], unrelated) + + def test_rows_without_instances_and_the_summary_pass_through(self) -> None: + file_row = { + "file": "a.sh", + "function": None, + "lane": "bash", + "values": {"lines": 3}, + "collector": "scc", + } + doc = document(file_row, pair((A, 1, 41), (B, 1, 41))) + doc["summary"] = {"files": 9, "functions": 0, "over_reference": {}} + result = run(doc) + out = json.loads(result.stdout) + self.assertEqual(out["measures"][0], file_row) + self.assertEqual(out["summary"], doc["summary"]) + self.assertEqual(out["excluded"], []) + + def test_stdin_that_is_not_json_is_a_usage_error(self) -> None: + result = run("not json") + self.assertEqual(result.returncode, 2) + self.assertIn("not a JSON document", result.stderr) + + def test_an_argument_is_a_usage_error(self) -> None: + self.assertEqual(run(document(), "--root").returncode, 2) + + +if __name__ == "__main__": + unittest.main() From 668e2423178977f7f2469ceff868680e133df237 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 16:16:13 +0000 Subject: [PATCH 07/15] feat(code-metrics): registry cluster lines for canonical-plus-copies classes Phase 3 of the duplication audit plan, committed alone because the registry edit fans CI's test selection out. - registry-filter.py reads a second line shape, ` -> ...`: a root-relative canonical copy and the plugin paths or gitignore-style globs that carry it. A clone group is excluded when every instance is the canonical or matches a member and the instances' directories are pairwise distinct. Instance paths are compared root-relative, so a run from a subdirectory matches the same lines, and the first matching line in file order wins. A plain line is still one path taken whole. - check-cross-plugin-source-drift.sh skips ` -> ` lines, which key on a root path this check never sees; its test covers the skip. - scripts/cross-plugin-source-registry.txt gains five annotated cluster lines for the sync-declared root lib/ canonicals: hook-utils, rewrite-guard, index-regen, resolve-convention-pattern, and parse-concern-value. The whole-tree audit now excludes each class (18, 7, 2, 2, and 4 instances) from the root and from a subdirectory. - config.md and the claude-config exclusion-set reference describe the grammar; the fixture registry carries a commented example. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01QShQwS5yDYYToBmofdEY5N --- .../code-metrics-duplication-audit/PLAN.md | 14 ++- .../audit-pass/reference/exclusion-set.md | 6 +- plugins/code-metrics/reference/config.md | 2 +- .../scripts/fixtures/registry/cluster.txt | 12 +- .../scripts/registry-filter.py | 109 +++++++++++++----- .../scripts/test_registry_filter.py | 108 +++++++++++++++++ scripts/check-cross-plugin-source-drift.sh | 8 ++ .../check-cross-plugin-source-drift.test.sh | 31 +++++ scripts/cross-plugin-source-registry.txt | 27 ++++- 9 files changed, 278 insertions(+), 39 deletions(-) diff --git a/docs/topics/code-metrics-duplication-audit/PLAN.md b/docs/topics/code-metrics-duplication-audit/PLAN.md index 2b7de94670..6b500f8c16 100644 --- a/docs/topics/code-metrics-duplication-audit/PLAN.md +++ b/docs/topics/code-metrics-duplication-audit/PLAN.md @@ -94,8 +94,12 @@ this repository the audit reads clean apart from genuine duplication. cluster lines present, each under its own annotation block, and its tests cover a marked line being skipped. - After this change, `audit-duplication.sh --json --registry scripts/cross-plugin-source-registry.txt --all` - on this repository reports zero surviving groups whose instances include a file under root `lib/`, - from the repository root and from a subdirectory alike. + on this repository reports zero surviving groups whose instances include a root `lib/` file that + a sync script declares as its canonical copy (`scripts/sync-*.sh --print-manifest` `src`), from + the repository root and from a subdirectory alike. Groups over the per-suite test-harness + boilerplate that `lib/*.test.sh` files share with plugin test files survive: no sync script + declares them and the shell-test-helpers convention keeps those helpers per file, so they are + the audit's finding, not sanctioned replication. - WHILE no registry is configured and none is passed, the report's `excluded[]` is empty and the summary states that no registry was configured. - The markdown Measures table for a duplication document lists clone groups in descending order of @@ -350,7 +354,7 @@ Review: code-design `grep -c '^/' plugins/code-metrics/scripts/fixtures/tool-output/jscpd-aligned3.json` prints `0`. - `bash plugins/code-metrics/scripts/dispatch.test.sh` exits 0 unchanged. -### Phase 3: Registry cluster lines, drift checker, this repo's five lines [TODO] +### Phase 3: Registry cluster lines, drift checker, this repo's five lines [DONE] Review: code-design @@ -411,8 +415,8 @@ Committed on its own (the registry edit fans CI's test selection out to roughly `bash scripts/check-cross-plugin-source-drift.test.sh` exits 0 (including the production-registry policy case). - Runtime probe (SKIP when absent) with jscpd 5.2.0 on PATH, from the repository root: - `audit-duplication.sh --json --registry scripts/cross-plugin-source-registry.txt --all | jq '[.measures[]|select(any(.instances[]; .file|test("^lib/")))]|length'` - prints `0`, and `jq '[.excluded[]|select(.path|startswith("lib/hook-utils.sh"))]|length'` + `audit-duplication.sh --json --registry scripts/cross-plugin-source-registry.txt --all | jq '[.measures[]|select(any(.instances[]; .file|test("^lib/[^/]*[.]sh$") and (.file|test("[.]test[.]sh$")|not)))]|length'` + prints `0` (the surviving `lib/` groups are all `lib/*.test.sh` harness boilerplate), and `jq '[.excluded[]|select(.path|startswith("lib/hook-utils.sh"))]|length'` prints `1` with that entry's `instances` length equal to `$(scripts/sync-hook-utils.sh --print-manifest | grep -c copy) + 1`; the same two commands run from `plugins/code-metrics` with `--registry ../../scripts/cross-plugin-source-registry.txt` diff --git a/plugins/claude-config/skills/audit-pass/reference/exclusion-set.md b/plugins/claude-config/skills/audit-pass/reference/exclusion-set.md index 97fcec64ee..1076bf72ec 100644 --- a/plugins/claude-config/skills/audit-pass/reference/exclusion-set.md +++ b/plugins/claude-config/skills/audit-pass/reference/exclusion-set.md @@ -15,8 +15,10 @@ dedicated script. Editing one copy breaks the sync path; a fix-capable pass that corrupt the cluster. **Derivation.** Ask the target whether it documents a shared-source registry. In this marketplace -that is `scripts/cross-plugin-source-registry.txt`, whose entries are paths *within* each plugin; -resolve each entry against every plugin root to get the live copy set. When the target documents no +that is `scripts/cross-plugin-source-registry.txt`, whose plain entries are paths *within* each +plugin (resolve each against every plugin root to get the live copy set) and whose cluster lines, +` -> ...`, name a root-relative canonical copy and the plugin paths or globs that +carry it (the canonical and every match are the copy set). When the target documents no such registry, **this class is empty** — say so in `skipped` rather than inferring one from similarity, which would exclude files nobody registered. diff --git a/plugins/code-metrics/reference/config.md b/plugins/code-metrics/reference/config.md index 0f145f0e87..d42dada3a7 100644 --- a/plugins/code-metrics/reference/config.md +++ b/plugins/code-metrics/reference/config.md @@ -68,7 +68,7 @@ The third column is written by hand and is not derived from anything. A row whos | `duplication.min_tokens` | `50` | Passed to the clone collector | | `duplication.min_lines` | `5` | Passed to the clone collector | | `duplication.ignore` | `[]` | Collector ignore globs | -| `duplication.registries` | `[]` | Sanctioned-replication registries (one path-within-plugin per line); a clone whose every instance sits at a listed path is excluded, not suppressed | +| `duplication.registries` | `[]` | Sanctioned-replication registries. A plain line is one path-within-plugin, taken whole, spaces included: a clone is excluded, not suppressed, when every instance ends with that path and the copies sit in distinct carrying directories. A line ` -> ...` is a cluster line: the root-relative canonical copy, then the paths or gitignore-style globs that carry it; a clone is excluded when every instance is the canonical or matches a member and the instances' directories are pairwise distinct. Instance paths are compared root-relative, and the first matching line in file order wins | | `duplication.max_lines` | `null` | A file with more lines is left out of the clone scan and named in the lane's `partial` run row; `null` or `0` means no line cap, which is what jscpd 5, PMD CPD, and SonarQube ship. A number is a plugin-local guard, not an upstream convention | | `duplication.max_size` | `1mb` | A file larger than this is left out and named the same way; `0` means no cap. jscpd 5.0.7 sets 1mb as its parser guard and SonarJS 1000kb for generated code. `kb` and `mb` are binary (1mb is 1,048,576 bytes); a CRLF checkout counts one more byte per line | | `duplication.rollup_depth` | `2` | Directory depth to which the markdown report lists per-directory rollup rows; the JSON carries every directory | diff --git a/plugins/code-metrics/scripts/fixtures/registry/cluster.txt b/plugins/code-metrics/scripts/fixtures/registry/cluster.txt index fdcc0ccc5b..5e064b56c8 100644 --- a/plugins/code-metrics/scripts/fixtures/registry/cluster.txt +++ b/plugins/code-metrics/scripts/fixtures/registry/cluster.txt @@ -2,8 +2,14 @@ # the shape a consuming repository declares its own deliberate replication # (this repository's scripts/cross-plugin-source-registry.txt is the live # example): one path-within-plugin per line, `#` comments and blank lines -# ignored. The cluster under fixtures/sources/cluster carries this path -# byte-identical in alpha and beta on purpose, so a clone over the two copies -# is an exclusion rather than duplication debt. +# ignored, or a cluster line ` -> ...` naming a +# root-relative canonical copy and the paths or globs that carry it. The +# cluster under fixtures/sources/cluster carries this path byte-identical in +# alpha and beta on purpose, so a clone over the two copies is an exclusion +# rather than duplication debt. shared/shared-utils.sh + +# The same cluster written as a cluster line, with a root canonical; left as a +# comment because the fixture tree carries no root copy. +# lib/shared-utils.sh -> plugins/*/shared/shared-utils.sh diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py b/plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py index 94c9c982d8..bc6fab2165 100755 --- a/plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py +++ b/plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py @@ -10,20 +10,33 @@ of a finding, so no suppression record is involved and the excluded groups stay visible in the document. -A registry is a text file with one path-within-plugin per line, `#` comments -and blank lines ignored (this repository's own -`scripts/cross-plugin-source-registry.txt` is the shape). A clone group is -dropped when one registry line accounts for EVERY instance: each instance's -path, relative to `--root` and written with forward slashes, is that line or -ends with `/` plus that line, and the prefixes in front of that suffix are all -distinct, so the copies sit in different carrying directories. Two clones -inside one directory are ordinary duplication and stay. +A registry is a text file, `#` comments and blank lines ignored (this +repository's own `scripts/cross-plugin-source-registry.txt` is the shape), with +two kinds of line: + +- A plain line is one path-within-plugin, taken whole, spaces included. It + sanctions a group when EVERY instance's path is that line or ends with `/` + plus that line, and the prefixes in front of that suffix are all distinct, + so the copies sit in different carrying directories. +- A line containing ` -> ` is a cluster line: the text before the arrow is the + root-relative canonical path, the whitespace-separated tokens after it are + members, each a literal root-relative path or a gitignore-style glob + (`plugins/*/hooks/hook-utils.sh`, matched by the plugin's `pathglob`). It + sanctions a group when every instance is the canonical path or matches one + member, and the instances' directories are pairwise distinct. + +Two clones inside one directory are ordinary duplication under either rule and +stay. Every instance path is compared root-relative: a cwd-relative path is +joined onto the working directory and taken relative to `--root`, so a run from +a subdirectory (where the dispatcher names files `../../lib/x.sh`) matches the +same lines a run from the root does. Lines are tried in file order and the +first matching line wins. Each dropped group is appended to `excluded[]` as `{"registry", "line", "path", "instances"}`, naming the registry file, the 1-based line number, and -the line's text that sanctioned it. Rows without `instances` pass through -untouched, and `summary` is left alone: the caller recomputes it with -`report.py resummarize`. +the line's text that sanctioned it (for a cluster line, the whole line). Rows +without `instances` pass through untouched, and `summary` is left alone: the +caller recomputes it with `report.py resummarize`. `--zero-floor` is the pass the caller runs AFTER that recomputation, with no registries: it states `duplicated_lines: 0` and `clone_groups: 0` when a @@ -41,41 +54,58 @@ import json import os import sys +from pathlib import Path from typing import Any +sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "scripts")) +from pathglob import matches as glob_matches # noqa: E402 + MIN_PYTHON = (3, 9) +CLUSTER_MARKER = " -> " + +# (line number, line text, canonical path or plain token, members). A plain +# line has no members. +Entry = tuple[int, str, str, list[str]] + + +def _clean(path: str) -> str: + return path.replace("\\", "/").lstrip("/") -def read_registry(path: str) -> list[tuple[int, str]]: - entries: list[tuple[int, str]] = [] +def read_registry(path: str) -> list[Entry]: + entries: list[Entry] = [] with open(path, encoding="utf-8") as handle: for number, raw in enumerate(handle, 1): line = raw.strip() if not line or line.startswith("#"): continue - entries.append((number, line.replace("\\", "/").lstrip("/"))) + if CLUSTER_MARKER in line: + canonical, _, rest = line.partition(CLUSTER_MARKER) + members = [_clean(token) for token in rest.split()] + entries.append((number, line, _clean(canonical.strip()), members)) + else: + entries.append((number, line, _clean(line), [])) return entries def relative(path: str, root: str) -> str: + """The instance path root-relative, with forward slashes.""" path = (path or "").replace("\\", "/") - if os.path.isabs(path) and root: + if root: + absolute = path if os.path.isabs(path) else os.path.join(os.getcwd(), path) try: - path = os.path.relpath(path, root).replace("\\", "/") + path = os.path.relpath(absolute, root).replace("\\", "/") except ValueError: - return path + pass while path.startswith("./"): path = path[2:] return path -def sanctions(entry: str, instances: list[dict[str, Any]], root: str) -> bool: - """True when this registry line accounts for every instance of the group.""" - if len(instances) < 2: - return False +def sanctions_plain(entry: str, paths: list[str]) -> bool: + """True when this plain line accounts for every instance of the group.""" prefixes = set() - for instance in instances: - path = relative(str(instance.get("file", "")), root) + for path in paths: if path == entry: prefix = "" elif path.endswith("/" + entry): @@ -88,9 +118,34 @@ def sanctions(entry: str, instances: list[dict[str, Any]], root: str) -> bool: return True +def sanctions_cluster(canonical: str, members: list[str], paths: list[str]) -> bool: + """True when this cluster line accounts for every instance of the group.""" + directories = set() + for path in paths: + if path != canonical and not any( + glob_matches(member, path) for member in members + ): + return False + directory = os.path.dirname(path) + if directory in directories: + return False + directories.add(directory) + return True + + +def sanctions(entry: Entry, instances: list[dict[str, Any]], root: str) -> bool: + if len(instances) < 2: + return False + paths = [relative(str(instance.get("file", "")), root) for instance in instances] + _, _, token, members = entry + if members: + return sanctions_cluster(token, members, paths) + return sanctions_plain(token, paths) + + def filter_document( document: dict[str, Any], - registries: list[tuple[str, list[tuple[int, str]]]], + registries: list[tuple[str, list[Entry]]], root: str, ) -> dict[str, Any]: kept: list[dict[str, Any]] = [] @@ -100,9 +155,9 @@ def filter_document( match = None if instances: for registry_path, entries in registries: - for number, entry in entries: + for entry in entries: if sanctions(entry, instances, root): - match = (registry_path, number, entry) + match = (registry_path, entry[0], entry[1]) break if match: break @@ -151,7 +206,7 @@ def main(argv: list[str]) -> int: parser.add_argument("--zero-floor", action="store_true") args = parser.parse_args(argv) - registries: list[tuple[str, list[tuple[int, str]]]] = [] + registries: list[tuple[str, list[Entry]]] = [] for path in args.registry: if not os.path.isfile(path): print(f"registry-filter.py: registry not found: {path}", file=sys.stderr) diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py b/plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py index 4cc5d2ea4b..9738f206ad 100755 --- a/plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py +++ b/plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py @@ -147,6 +147,114 @@ def test_the_committed_cluster_registry_declares_the_fixture_path(self) -> None: self.assertEqual(out["measures"], []) self.assertEqual(out["excluded"][0]["path"], "shared/shared-utils.sh") + def write_registry(self, text: str) -> Path: + path = Path(self.tmp.name) / "clusters.txt" + path.write_text(text, encoding="utf-8") + return path + + def test_a_cluster_line_excludes_the_canonical_plus_its_copies(self) -> None: + registry = self.write_registry( + "# canonical and copies\nlib/hook-utils.sh -> plugins/*/hooks/hook-utils.sh\n" + ) + doc = document( + [ + "lib/hook-utils.sh", + "plugins/one/hooks/hook-utils.sh", + "plugins/two/hooks/hook-utils.sh", + ] + ) + result = run(doc, "--root", ".", "--registry", str(registry)) + self.assertEqual(result.returncode, 0, result.stderr) + out = json.loads(result.stdout) + self.assertEqual(out["measures"], []) + entry = out["excluded"][0] + self.assertEqual( + entry["path"], "lib/hook-utils.sh -> plugins/*/hooks/hook-utils.sh" + ) + self.assertEqual(entry["line"], 2) + self.assertEqual(len(entry["instances"]), 3) + + def test_a_glob_member_matches_and_a_stranger_keeps_the_group(self) -> None: + registry = self.write_registry( + "lib/parse.sh -> plugins/*/skills/*/scripts/parse.sh " + "plugins/*/skills/*/scripts/lib/parse.sh\n" + ) + sanctioned = document( + [ + "lib/parse.sh", + "plugins/a/skills/x/scripts/parse.sh", + "plugins/b/skills/y/scripts/lib/parse.sh", + ] + ) + result = run(sanctioned, "--root", ".", "--registry", str(registry)) + self.assertEqual(json.loads(result.stdout)["measures"], []) + stranger = document(["lib/parse.sh", "plugins/a/hooks/parse.sh"]) + result = run(stranger, "--root", ".", "--registry", str(registry)) + out = json.loads(result.stdout) + self.assertEqual(len(out["measures"]), 1) + self.assertEqual(out["excluded"], []) + + def test_two_cluster_instances_in_one_directory_keep_the_group(self) -> None: + registry = self.write_registry("lib/a.sh -> plugins/*/hooks/*.sh\n") + doc = document(["lib/a.sh", "plugins/one/hooks/a.sh", "plugins/one/hooks/b.sh"]) + result = run(doc, "--root", ".", "--registry", str(registry)) + out = json.loads(result.stdout) + self.assertEqual(len(out["measures"]), 1) + self.assertEqual(out["excluded"], []) + + def test_a_plain_line_with_a_space_is_one_path(self) -> None: + registry = self.write_registry("hooks/shared file.sh\n") + doc = document( + ["plugins/one/hooks/shared file.sh", "plugins/two/hooks/shared file.sh"] + ) + result = run(doc, "--root", ".", "--registry", str(registry)) + out = json.loads(result.stdout) + self.assertEqual(out["measures"], []) + self.assertEqual(out["excluded"][0]["path"], "hooks/shared file.sh") + + def test_the_first_matching_line_in_file_order_wins(self) -> None: + registry = self.write_registry("lib/x.sh -> plugins/*/hooks/x.sh\nhooks/x.sh\n") + doc = document(["plugins/one/hooks/x.sh", "plugins/two/hooks/x.sh"]) + result = run(doc, "--root", ".", "--registry", str(registry)) + entry = json.loads(result.stdout)["excluded"][0] + self.assertEqual( + (entry["path"], entry["line"]), ("lib/x.sh -> plugins/*/hooks/x.sh", 1) + ) + reversed_registry = self.write_registry( + "hooks/x.sh\nlib/x.sh -> plugins/*/hooks/x.sh\n" + ) + result = run(doc, "--root", ".", "--registry", str(reversed_registry)) + entry = json.loads(result.stdout)["excluded"][0] + self.assertEqual((entry["path"], entry["line"]), ("hooks/x.sh", 1)) + + def test_cwd_relative_instances_from_a_subdirectory_still_match(self) -> None: + root = Path(self.tmp.name) + sub = root / "plugins" / "code-metrics" + sub.mkdir(parents=True) + registry = self.write_registry( + "lib/hook-utils.sh -> plugins/*/hooks/hook-utils.sh\n" + ) + doc = document(["../../lib/hook-utils.sh", "../one/hooks/hook-utils.sh"]) + result = subprocess.run( + [ + sys.executable, + str(SCRIPT), + "--root", + str(root), + "--registry", + str(registry), + ], + input=json.dumps(doc), + capture_output=True, + text=True, + cwd=sub, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + out = json.loads(result.stdout) + self.assertEqual(out["measures"], []) + self.assertEqual(len(out["excluded"]), 1) + def test_the_zero_floor_states_zero_once_a_collector_ran(self) -> None: doc = document() doc["run"] = [{"lane": "bash", "measure": "duplication", "status": "ok"}] diff --git a/scripts/check-cross-plugin-source-drift.sh b/scripts/check-cross-plugin-source-drift.sh index 8c0f701644..a2ed10fab1 100755 --- a/scripts/check-cross-plugin-source-drift.sh +++ b/scripts/check-cross-plugin-source-drift.sh @@ -154,6 +154,14 @@ if [[ -f "$registry" ]]; then line="${line#"${line%%[![:space:]]*}"}" line="${line%"${line##*[![:space:]]}"}" [[ -z "$line" ]] && continue + # A line containing ` -> ` is a cluster line (` -> ...`, + # a root-relative canonical copy and the plugin paths or globs that carry + # it). It belongs to the duplication audit's reader + # (plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py), + # which excludes the whole class from its clone count; this check keys + # clusters by path-within-plugin, so the line is neither registered here + # nor reported stale. + [[ "$line" == *" -> "* ]] && continue registered["$line"]=1 done <"$registry" fi diff --git a/scripts/check-cross-plugin-source-drift.test.sh b/scripts/check-cross-plugin-source-drift.test.sh index 1e59f548dd..a04d9c0a6b 100755 --- a/scripts/check-cross-plugin-source-drift.test.sh +++ b/scripts/check-cross-plugin-source-drift.test.sh @@ -199,6 +199,37 @@ else fi rm -rf "$f" +# --- a cluster line (` -> `) is the duplication audit's, not this check's --- +# +# ` -> ...` names a root-relative canonical copy and the +# plugin paths or globs that carry it, for registry-filter.py in the +# code-metrics plugin. This check keys clusters by path-within-plugin, so the +# line must be skipped: registering it would report it REGISTRY STALE on every +# run, since no plugin carries a path spelled `lib/... -> ...`. +f="$(new_fixture)" +plugin_file "$f" alpha hooks/shared.sh "same" +plugin_file "$f" beta hooks/shared.sh "same" +registry "$f" "hooks/shared.sh" "lib/shared.sh -> plugins/*/hooks/shared.sh" +if out="$(run_check "$f" 2>&1)"; then + if grep -q 'REGISTRY STALE' <<<"$out"; then + fail "a cluster line must not be reported stale, got: $out" + else + ok "--check skips a cluster line instead of registering it" + fi +else + fail "--check should pass with a cluster line beside a registered path, got: $out" +fi +if out="$(run_discover "$f" 2>&1)"; then + if grep -q ' -> ' <<<"$out"; then + fail "discover must not list a cluster line, got: $out" + else + ok "discover leaves a cluster line out of the inventory" + fi +else + fail "discover should exit 0 with a cluster line in the registry, got: $out" +fi +rm -rf "$f" + # --- production registry: every cluster documents its enforcement path (#2404) - REGISTRY="$SELF_DIR/cross-plugin-source-registry.txt" if [[ ! -f "$REGISTRY" ]]; then diff --git a/scripts/cross-plugin-source-registry.txt b/scripts/cross-plugin-source-registry.txt index 28b876da08..c3279495c1 100644 --- a/scripts/cross-plugin-source-registry.txt +++ b/scripts/cross-plugin-source-registry.txt @@ -9,14 +9,26 @@ # (already a required CI job) and names its canonical copy instead, so a drift # failure says which direction to fix. # -# One path-within-plugin per line, relative to each plugin's own root. +# One path-within-plugin per line, relative to each plugin's own root. A line +# of the form ` -> ...` is a cluster line: the root-relative +# canonical copy, then the plugin paths or globs that carry it. It is read only +# by the code-metrics duplication audit (registry-filter.py), which excludes +# the canonical plus every copy as one sanctioned class; this script skips it. # Dedicated check: scripts/sync-hook-utils.sh --check (CI: hook-utils-sync) hooks/hook-utils.sh +# Dedicated check: scripts/sync-hook-utils.sh --check (CI: hook-utils-sync). +# Cluster line for the duplication audit: the root canonical and the copies. +lib/hook-utils.sh -> plugins/*/hooks/hook-utils.sh + # Dedicated check: scripts/sync-rewrite-guard.sh --check (CI: rewrite-guard-sync) hooks/rewrite-guard.sh +# Dedicated check: scripts/sync-rewrite-guard.sh --check (CI: rewrite-guard-sync). +# Cluster line for the duplication audit: the root canonical and the copies. +lib/rewrite-guard.sh -> plugins/*/hooks/rewrite-guard.sh + # Dedicated check: scripts/validate-plugin-contracts.mjs (lifecycleProtocolCopies) reference/artifact-protocol.md @@ -36,6 +48,19 @@ lib/state-key.sh # second plugin consumes it. # scripts/index-regen.sh +# Dedicated check: scripts/sync-index-regen.sh --check. Cluster line for the +# duplication audit: the root canonical and its copy are one class today, so +# the line is live even while the path-within-plugin entry above stays out. +lib/index-regen.sh -> plugins/*/scripts/index-regen.sh + +# Dedicated check: scripts/sync-resolve-convention-pattern.sh --check. +# Cluster line for the duplication audit: the root canonical and its copy. +lib/resolve-convention-pattern.sh -> plugins/*/hooks/resolve-convention-pattern.sh + +# Dedicated check: scripts/sync-parse-concern-value.sh --check. Cluster line +# for the duplication audit: the copies sit at two shapes within a plugin. +lib/parse-concern-value.sh -> plugins/*/skills/*/scripts/parse-concern-value.sh plugins/*/skills/*/scripts/lib/parse-concern-value.sh + # Dedicated check: scripts/sync-spawn-noise.sh --check (CI: spawn-noise-sync) lib/spawn_noise.py From 2c2866e7f9c43dfc29c84b4c5654e293a020d477 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 16:35:50 +0000 Subject: [PATCH 08/15] feat(code-metrics): duplication rollups, largest-first sort, clearer summary Phase 4 of the duplication audit plan. - report.py summarize adds `summary.by_lane` and `summary.by_directory` (every ancestor of each group's first instance, cumulative, made relative to --root) beside duplicated_lines and clone_groups; by_directory["."] and the per-lane sum both restate the totals. The zero floor states both as empty maps on a clone-free run. - assemble reads a `partial` run row with no measure rows as `partial`, not `empty`: a lane that skipped every file said so in its row. - render lists a duplication document's groups largest first, adds a `## Rollup` section (per lane, and per directory to --rollup-depth, default 2 from duplication.rollup_depth), summarizes as `Files with clones: N.`, states an empty excluded list with its reason, adds a `Partial:` line, and prints one headline with the install hint when no clone detector ran in any lane. Every other skill's document renders as before. - report-schema.md documents the rollups, the ignore-unknown-keys rule, the partial status for a skipped lane, and intentional clones. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01QShQwS5yDYYToBmofdEY5N --- .../code-metrics-duplication-audit/PLAN.md | 6 +- .../code-metrics/reference/report-schema.md | 22 +- plugins/code-metrics/scripts/report.py | 220 +++++++++++--- plugins/code-metrics/scripts/test_report.py | 278 ++++++++++++++++++ .../scripts/audit-duplication.sh | 12 +- .../scripts/registry-filter.py | 4 + .../scripts/test_registry_filter.py | 8 + 7 files changed, 507 insertions(+), 43 deletions(-) diff --git a/docs/topics/code-metrics-duplication-audit/PLAN.md b/docs/topics/code-metrics-duplication-audit/PLAN.md index 6b500f8c16..0965a7d8f3 100644 --- a/docs/topics/code-metrics-duplication-audit/PLAN.md +++ b/docs/topics/code-metrics-duplication-audit/PLAN.md @@ -422,7 +422,7 @@ Committed on its own (the registry edit fans CI's test selection out to roughly from `plugins/code-metrics` with `--registry ../../scripts/cross-plugin-source-registry.txt` print the same values. -### Phase 4: Report sort, rollups, additive summary fields, summary line, no-detector headline [TODO] +### Phase 4: Report sort, rollups, additive summary fields, summary line, no-detector headline [DONE] Review: code-design @@ -474,7 +474,9 @@ Review: code-design `audit-size.sh --all | grep -c 'Functions:'` prints `1`. - `audit-duplication.sh --json --all | jq '([.summary.by_lane[].duplicated_lines]|add) == .summary.duplicated_lines and .summary.by_directory["."].duplicated_lines == .summary.duplicated_lines'` prints `true`. -- `PATH= audit-duplication.sh --all | grep -c 'npm install -g jscpd'` prints `1`. +- `PATH= audit-duplication.sh --all | grep -c 'No clone detector ran'` prints `1`; + the install hint appears in that headline and again in each lane row's reason, which the + dispatcher builds with the hint embedded and this phase leaves unchanged. ### Phase 5: SKILL.md, README, CHANGELOG, version, dogfood [TODO] diff --git a/plugins/code-metrics/reference/report-schema.md b/plugins/code-metrics/reference/report-schema.md index bfc1144429..9c2f28f142 100644 --- a/plugins/code-metrics/reference/report-schema.md +++ b/plugins/code-metrics/reference/report-schema.md @@ -11,15 +11,31 @@ read, so its shape is stable within the `v1` schema string. | `schema` | string | `code-metrics/v1` | | `skill` | string | The producing skill, for example `audit-size` | | `generated_at` | string | UTC timestamp, `YYYY-MM-DDTHH:MM:SSZ` | -| `status` | string | `complete` (every implied lane and measure ran; a `not-applicable` row implies nothing and never withholds it), `partial` (at least one `unavailable`, `deferred`, or `partial` row), `empty` (nothing was measured; the markdown headline reads "Measured nothing") | +| `status` | string | `complete` (every implied lane and measure ran; a `not-applicable` row implies nothing and never withholds it), `partial` (at least one `unavailable`, `deferred`, or `partial` row; a lane that skipped every file is `partial` even with no measure row, because its run row states the skip), `empty` (nothing was measured; the markdown headline reads "Measured nothing") | | `scope` | object | `mode` (`change`, `paths`, `all`), `base` (the merge-base's short SHA under `change`, else `null`), `files` (count in scope), `unclassified` (how many of those belong to no lane, so `files` minus `unclassified` is the measured count), `excluded` (count dropped by scope exclusions) | | `run` | array | The "Coverage of this run" table, one row per lane and measure the scope implied | | `thresholds` | array | The references in force: `measure`, `reference` (number or `null`), `provenance`, `layer` (which config layer supplied it, or `bundled default`) | | `measures` | array | The rows, see below | -| `summary` | object | `files`, `functions`, `over_reference` (measure name to count); when clone-group rows are present, `duplicated_lines` (sum of each group's `values.lines`, one group counted once, after registry exclusions) and `clone_groups` | -| `excluded` | array | Duplication only: clone groups dropped by a sanctioned-replication registry, each naming the registry path and line | +| `summary` | object | `files`, `functions`, `over_reference` (measure name to count); when clone-group rows are present, `duplicated_lines` (sum of each group's `values.lines`, one group counted once, after registry exclusions), `clone_groups`, `by_lane`, and `by_directory` (see below) | +| `excluded` | array | Duplication only: clone groups dropped by a sanctioned-replication registry (intentional clones the repository declares about itself), each naming the registry path and line | | `unavailable` | array | `lane/measure` strings for every `run` row whose status is `unavailable` | +A reader ignores keys it does not know: fields are added within `v1` (the rollups and the run +row's `hint` were), never renamed or removed. + +## Duplication rollups + +`summary.by_lane` maps each lane to `{"groups", "duplicated_lines"}` over the surviving clone +groups, and `summary.by_directory` maps `.` and every ancestor directory of each group's first +instance (instances are sorted by path, so that is the lowest path; paths are made relative to the +repository root) to the same shape. A group counts once under every ancestor, so a parent includes +its children and directory rows cannot be summed; two identities hold instead: +`by_directory["."].duplicated_lines == summary.duplicated_lines` and the `by_lane` values sum to +it. A duplication run that found or kept no group carries both as empty maps, beside its +`duplicated_lines: 0`; a document without clone-group rows and without a duplication collector run +carries neither. The markdown `## Rollup` section lists directories to `duplication.rollup_depth` +(default 2); the JSON carries every directory. + `summary.functions` counts functions, not rows: one function measured by two collectors produces two rows and counts once. Rows are grouped by file and name, and a group counts as many functions as it has distinct `start_line` values, or as one when no row in it reports a start line. So two `render` diff --git a/plugins/code-metrics/scripts/report.py b/plugins/code-metrics/scripts/report.py index d459156efe..52862d07ee 100755 --- a/plugins/code-metrics/scripts/report.py +++ b/plugins/code-metrics/scripts/report.py @@ -10,20 +10,28 @@ report.py assemble --skill --scope --run --measures --thresholds - [--excluded ] + [--excluded ] [--root ] Print the report document: `run[]` is the coverage-of-this-run table, `measures[]` gains `over_reference`, `summary` counts, `unavailable[]` lists every non-ok lane/measure, and `status` is complete, partial, or - empty. A value that was not measured is `null`, never zero. + empty. A value that was not measured is `null`, never zero. A `partial` + run row counts as measured, so a lane that skipped every file is + `partial`, not `empty`. - report.py render [< report.json] - Print the markdown rendering of a report document read from stdin. + report.py render [--rollup-depth ] [< report.json] + Print the markdown rendering of a report document read from stdin. A + duplication document (clone-group rows, or `skill` audit-duplication) + lists groups largest first, adds a `## Rollup` section with per-lane and + per-directory tables (directories to `--rollup-depth`, default 2), and + summarizes as `Files with clones`; every other document renders as it + always has. - report.py resummarize [< report.json] + report.py resummarize [--root ] [< report.json] Recompute `summary` from `measures[]` and print the document; for a skill that drops rows after assembly (a duplication registry moving clone groups into `excluded[]`). Clone-group rows (`instances[]`) add - `summary.duplicated_lines` and `summary.clone_groups`. + `summary.duplicated_lines`, `summary.clone_groups`, `summary.by_lane`, + and `summary.by_directory` (paths made relative to `--root`). Exit 0 on success, 2 on a usage error or unreadable input. """ @@ -33,6 +41,7 @@ import argparse import datetime as _dt import json +import os import sys from typing import Any @@ -107,14 +116,49 @@ def _over(threshold: dict[str, Any], value: Any) -> bool: return value >= reference -def summarize(measures: list[dict[str, Any]]) -> dict[str, Any]: +def _root_relative(path: str, root: str) -> str: + """The path relative to `root` with forward slashes; unchanged without a root.""" + path = (path or "").replace("\\", "/") + if root: + absolute = path if os.path.isabs(path) else os.path.join(os.getcwd(), path) + try: + path = os.path.relpath(absolute, root).replace("\\", "/") + except ValueError: + pass + while path.startswith("./"): + path = path[2:] + return path + + +def _ancestors(path: str) -> list[str]: + """`.` and every directory above the file, root first.""" + parts = path.split("/")[:-1] + return ["."] + ["/".join(parts[: index + 1]) for index in range(len(parts))] + + +def _tally(buckets: dict[str, dict[str, int]], key: str, lines: int) -> None: + bucket = buckets.setdefault(key, {"groups": 0, "duplicated_lines": 0}) + bucket["groups"] += 1 + bucket["duplicated_lines"] += lines + + +def summarize(measures: list[dict[str, Any]], root: str = "") -> dict[str, Any]: """The `summary` block, derived from `measures[]` alone so a skill that drops rows after assembly (a duplication registry exclusion) can recompute it through the `resummarize` verb. Counts use each row's `over_reference` list as assembled; clone-group rows (those carrying `instances[]`) add `duplicated_lines` (sum of `values.lines`, each group counted once) and - `clone_groups`, and their instance files count toward `files`.""" + `clone_groups`, and their instance files count toward `files`. + + Clone-group rows also add `by_lane` (lane to `{groups, duplicated_lines}`) + and `by_directory` (the same shape for `.` and every ancestor directory of + each group's first instance, made relative to `root`). A group counts once + per ancestor, so a parent includes its children and the rows cannot be + summed, while `by_directory["."]` and the per-lane sum both restate the + totals.""" files: set[str] = set() + by_lane: dict[str, dict[str, int]] = {} + by_directory: dict[str, dict[str, int]] = {} # (file, name) -> the distinct start lines reported for it. A name is not an # identity: one file can hold two `render` methods. A start line is not one # either, because a collector that reports Halstead for a function need not @@ -146,11 +190,17 @@ def summarize(measures: list[dict[str, Any]]) -> dict[str, Any]: if instances: clone_groups += 1 lines = (row.get("values") or {}).get("lines") + counted = 0 if isinstance(lines, (int, float)) and not isinstance(lines, bool): - duplicated_lines += int(lines) + counted = int(lines) + duplicated_lines += counted for instance in instances: if instance.get("file"): files.add(instance["file"]) + _tally(by_lane, str(row.get("lane") or "*"), counted) + first = _root_relative(str(instances[0].get("file") or ""), root) + for directory in _ancestors(first): + _tally(by_directory, directory, counted) summary: dict[str, Any] = { "files": len(files), "functions": sum(max(1, len(starts)) for starts in functions.values()), @@ -159,6 +209,8 @@ def summarize(measures: list[dict[str, Any]]) -> dict[str, Any]: if clone_groups: summary["duplicated_lines"] = duplicated_lines summary["clone_groups"] = clone_groups + summary["by_lane"] = by_lane + summary["by_directory"] = by_directory return summary @@ -169,6 +221,7 @@ def assemble( measures: list[dict[str, Any]], threshold_entries: list[dict[str, Any]], excluded: list[dict[str, Any]], + root: str = "", ) -> dict[str, Any]: for row in run: if row.get("status") not in RUN_STATUSES: @@ -186,10 +239,13 @@ def assemble( # exist for that lane), so it never withholds `complete`; `unavailable`, # `deferred` and `partial` rows do, because something implied was not # measured. `partial` still counts as having produced rows, so a run that - # measured part of a lane reads as `partial` rather than as `empty`. + # measured part of a lane reads as `partial` rather than as `empty`, even + # when it skipped every file and has no row to show: the skip is stated in + # the run row, and "Measured nothing" would contradict it. ok_rows = [row for row in run if row.get("status") in ("ok", "partial")] + partial_rows = [row for row in run if row.get("status") == "partial"] settled = [row for row in run if row.get("status") in ("ok", "not-applicable")] - if not ok_rows or not measures: + if not ok_rows or (not measures and not partial_rows): status = "empty" elif len(settled) == len(run): status = "complete" @@ -209,7 +265,7 @@ def assemble( for entry in threshold_entries ], "measures": measures, - "summary": summarize(measures), + "summary": summarize(measures, root), "excluded": excluded, "unavailable": [ f"{row.get('lane', '*')}/{row.get('measure', '*')}" @@ -227,13 +283,55 @@ def _fmt(value: Any) -> str: return str(value) -def render(doc: dict[str, Any]) -> str: +def _is_duplication(doc: dict[str, Any]) -> bool: + return doc.get("skill") == "audit-duplication" or any( + row.get("instances") for row in doc.get("measures", []) + ) + + +def _depth(directory: str) -> int: + return 0 if directory == "." else directory.count("/") + 1 + + +def _clone_sort_key(row: dict[str, Any]) -> tuple[int, int, str]: + values = row.get("values") or {} + instances = row.get("instances") or [{}] + + def number(value: Any) -> int: + return int(value) if isinstance(value, (int, float)) else 0 + + return ( + -number(values.get("lines")), + -number(values.get("tokens")), + str(instances[0].get("file") or ""), + ) + + +def render(doc: dict[str, Any], rollup_depth: int = 2) -> str: lines: list[str] = [] status = doc.get("status", "empty") headline = "Measured nothing" if status == "empty" else f"Status: {status}" scope = doc.get("scope", {}) + duplication = _is_duplication(doc) lines.append(f"# code-metrics: {doc.get('skill', '?')}") lines.append("") + detector_rows = [ + row for row in doc.get("run", []) if row.get("measure") == "duplication" + ] + if ( + duplication + and detector_rows + and all(row.get("status") == "unavailable" for row in detector_rows) + ): + # One headline for the whole run: the lane rows below still carry + # each probe's own reason, so this names the fix once, not per lane. + hint = next((row.get("hint") for row in detector_rows if row.get("hint")), "") + lines.append( + "No clone detector ran in any lane" + + (f": {hint}" if hint else "") + + ". Run `/code-metrics:setup` to install one." + ) + lines.append("") lines.append( f"{headline}. Scope: {scope.get('mode', '?')}" + (f" against `{scope['base']}`" if scope.get("base") else "") @@ -285,14 +383,20 @@ def render(doc: dict[str, Any]) -> str: lines.append(header) lines.append("|" + "---|" * (4 + len(keys))) shown = 0 - for row in sorted( - measures, - key=lambda r: ( - -len(r.get("over_reference", [])), - r.get("file", ""), - r.get("start_line") or 0, - ), - ): + if duplication: + # Largest group first: the reader's question is "what is the + # biggest copy", not which file sorts first. + ordered = sorted(measures, key=_clone_sort_key) + else: + ordered = sorted( + measures, + key=lambda r: ( + -len(r.get("over_reference", [])), + r.get("file", ""), + r.get("start_line") or 0, + ), + ) + for row in ordered: if shown >= MAX_RENDERED_ROWS: lines.append( f"| ... | | | {' | '.join('' for _ in keys)} | {len(measures) - shown} more rows in the JSON |" @@ -312,18 +416,52 @@ def render(doc: dict[str, Any]) -> str: ) shown += 1 summary = doc.get("summary", {}) + by_lane = summary.get("by_lane") or {} + by_directory = summary.get("by_directory") or {} + if duplication and (by_lane or by_directory): + lines.append("") + lines.append("## Rollup") + lines.append("") + lines.append("| Lane | Clone groups | Duplicated lines |") + lines.append("|---|---|---|") + for lane, bucket in sorted(by_lane.items()): + lines.append( + f"| {lane} | {bucket.get('groups', 0)} | {bucket.get('duplicated_lines', 0)} |" + ) + lines.append("") + lines.append( + f"| Directory (to depth {rollup_depth}) | Clone groups | Duplicated lines |" + ) + lines.append("|---|---|---|") + for directory, bucket in sorted(by_directory.items()): + if _depth(directory) <= rollup_depth: + lines.append( + f"| {directory} | {bucket.get('groups', 0)} | " + f"{bucket.get('duplicated_lines', 0)} |" + ) + lines.append("") + lines.append( + "A group is attributed to every directory above its first instance, so a parent " + "includes its children and the directory rows cannot be summed; `.` restates the " + "totals. The JSON carries every directory." + ) lines.append("") lines.append("## Summary") lines.append("") - lines.append( - f"Files: {summary.get('files', 0)}. Functions: {summary.get('functions', 0)}. " - + "Over reference: " - + ( - ", ".join(f"{k} {v}" for k, v in summary.get("over_reference", {}).items()) - or "none" + if duplication: + lines.append(f"Files with clones: {summary.get('files', 0)}.") + else: + lines.append( + f"Files: {summary.get('files', 0)}. Functions: {summary.get('functions', 0)}. " + + "Over reference: " + + ( + ", ".join( + f"{k} {v}" for k, v in summary.get("over_reference", {}).items() + ) + or "none" + ) + + "." ) - + "." - ) if "duplicated_lines" in summary: lines.append( f"Duplicated lines: {summary['duplicated_lines']} in " @@ -333,8 +471,20 @@ def render(doc: dict[str, Any]) -> str: lines.append( f"Excluded by a sanctioned-replication registry: {len(doc['excluded'])}." ) + elif duplication: + lines.append( + "Excluded by a sanctioned-replication registry: 0 (no registry configured, or " + "none matched)." + ) if doc.get("unavailable"): lines.append("Unavailable: " + ", ".join(doc["unavailable"]) + ".") + partial = [ + f"{row.get('lane', '*')}/{row.get('measure', '*')}" + for row in doc.get("run", []) + if row.get("status") == "partial" + ] + if duplication and partial: + lines.append("Partial: " + ", ".join(partial) + ".") return "\n".join(lines) + "\n" @@ -351,8 +501,11 @@ def main(argv: list[str]) -> int: p_asm.add_argument("--measures", required=True) p_asm.add_argument("--thresholds", required=True) p_asm.add_argument("--excluded") - sub.add_parser("render") - sub.add_parser("resummarize") + p_asm.add_argument("--root", default="") + p_render = sub.add_parser("render") + p_render.add_argument("--rollup-depth", type=int, default=2) + p_res = sub.add_parser("resummarize") + p_res.add_argument("--root", default="") args = parser.parse_args(argv) if args.command == "thresholds": config = _read_json(args.config) @@ -366,15 +519,16 @@ def main(argv: list[str]) -> int: _read_jsonl(args.measures), _read_json(args.thresholds), _read_jsonl(args.excluded), + args.root, ) print(json.dumps(doc, indent=2)) return 0 doc = json.load(sys.stdin) if args.command == "resummarize": - doc["summary"] = summarize(doc.get("measures", [])) + doc["summary"] = summarize(doc.get("measures", []), args.root) print(json.dumps(doc, indent=2)) return 0 - sys.stdout.write(render(doc)) + sys.stdout.write(render(doc, args.rollup_depth)) return 0 diff --git a/plugins/code-metrics/scripts/test_report.py b/plugins/code-metrics/scripts/test_report.py index 57c10dab42..6e7390fb25 100755 --- a/plugins/code-metrics/scripts/test_report.py +++ b/plugins/code-metrics/scripts/test_report.py @@ -687,5 +687,283 @@ def test_resummarize_recomputes_the_summary_after_rows_are_dropped(self) -> None self.assertEqual(out["run"], doc["run"]) +def clone_row(lane: str, first: str, second: str, lines: int, tokens: int = 90) -> dict: + return { + "file": None, + "function": None, + "lane": lane, + "instances": [ + {"file": first, "start_line": 1, "end_line": lines}, + {"file": second, "start_line": 1, "end_line": lines}, + ], + "values": {"lines": lines, "tokens": tokens}, + "over_reference": [], + } + + +def duplication_doc(measures: list[dict], **overrides: object) -> dict: + doc = { + "schema": "code-metrics/v1", + "skill": "audit-duplication", + "status": "complete", + "scope": {"mode": "all", "base": None, "files": 4, "excluded": 0}, + "run": [ + { + "lane": "bash", + "measure": "duplication", + "collector": "jscpd 5.2.0", + "status": "ok", + "reason": None, + "hint": None, + } + ], + "thresholds": [], + "measures": measures, + "summary": {"files": 0, "functions": 0, "over_reference": {}}, + "excluded": [], + "unavailable": [], + } + doc.update(overrides) + return doc + + +def resummarized(doc: dict, *args: str) -> dict: + result = run("resummarize", *args, stdin=json.dumps(doc)) + assert result.returncode == 0, result.stderr + return json.loads(result.stdout) + + +class DuplicationRollupTests(unittest.TestCase): + ROWS = [ + clone_row("bash", "a/x/u.sh", "b/x/u.sh", 20), + clone_row("python", "a/y/v.py", "c/y/v.py", 7, 30), + clone_row("bash", "a/x/w.sh", "d/x/w.sh", 5, 12), + ] + + def test_the_rollups_sum_to_the_totals_and_root_restates_them(self) -> None: + summary = resummarized(duplication_doc(self.ROWS))["summary"] + self.assertEqual(summary["duplicated_lines"], 32) + self.assertEqual( + summary["by_lane"], + { + "bash": {"groups": 2, "duplicated_lines": 25}, + "python": {"groups": 1, "duplicated_lines": 7}, + }, + ) + self.assertEqual( + sum(b["duplicated_lines"] for b in summary["by_lane"].values()), + summary["duplicated_lines"], + ) + self.assertEqual( + summary["by_directory"]["."], {"groups": 3, "duplicated_lines": 32} + ) + + def test_directory_rollups_are_cumulative_over_the_first_instance(self) -> None: + by_directory = resummarized(duplication_doc(self.ROWS))["summary"][ + "by_directory" + ] + # Every group's first instance sits under `a`, so `a` carries all three + # while its children split them; the second instances count nowhere. + self.assertEqual(by_directory["a"], {"groups": 3, "duplicated_lines": 32}) + self.assertEqual(by_directory["a/x"], {"groups": 2, "duplicated_lines": 25}) + self.assertEqual(by_directory["a/y"], {"groups": 1, "duplicated_lines": 7}) + self.assertNotIn("b", by_directory) + self.assertEqual( + set(by_directory), {".", "a", "a/x", "a/y"}, sorted(by_directory) + ) + + def test_root_makes_directory_keys_root_relative(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + rows = [ + clone_row( + "bash", + str(root / "plugins" / "p" / "u.sh"), + str(root / "plugins" / "q" / "u.sh"), + 9, + ) + ] + by_directory = resummarized(duplication_doc(rows), "--root", tmp)[ + "summary" + ]["by_directory"] + self.assertEqual(set(by_directory), {".", "plugins", "plugins/p"}) + + def test_a_document_without_clone_rows_carries_no_rollup_maps(self) -> None: + summary = resummarized(duplication_doc([]))["summary"] + self.assertNotIn("by_lane", summary) + self.assertNotIn("by_directory", summary) + + +class DuplicationRenderTests(unittest.TestCase): + def rendered(self, doc: dict, *args: str) -> str: + result = run("render", *args, stdin=json.dumps(doc)) + self.assertEqual(result.returncode, 0, result.stderr) + return result.stdout + + def test_clone_groups_render_largest_first(self) -> None: + rows = [ + clone_row("bash", "a/small.sh", "b/small.sh", 5, 12), + clone_row("bash", "a/big.sh", "b/big.sh", 20), + clone_row("bash", "a/mid.sh", "b/mid.sh", 5, 40), + ] + out = self.rendered(resummarized(duplication_doc(rows))) + self.assertLess(out.index("a/big.sh"), out.index("a/mid.sh")) + self.assertLess(out.index("a/mid.sh"), out.index("a/small.sh")) + + def test_the_rollup_section_cuts_directories_at_the_depth(self) -> None: + rows = [clone_row("bash", "a/x/z/u.sh", "b/u.sh", 20)] + doc = resummarized(duplication_doc(rows)) + self.assertIn("a/x/z", doc["summary"]["by_directory"]) + out = self.rendered(doc) + self.assertIn("## Rollup", out) + self.assertIn("| bash | 1 | 20 |", out) + self.assertIn("| . | 1 | 20 |", out) + self.assertIn("| a/x | 1 | 20 |", out) + self.assertNotIn("| a/x/z |", out) + self.assertIn("| a/x/z | 1 | 20 |", self.rendered(doc, "--rollup-depth", "3")) + + def test_the_summary_line_counts_files_with_clones(self) -> None: + rows = [clone_row("bash", "a/u.sh", "b/u.sh", 20)] + out = self.rendered(resummarized(duplication_doc(rows))) + self.assertIn("\nFiles with clones: 2.\n", out) + self.assertNotIn("Functions:", out) + + def test_an_empty_excluded_list_is_stated_with_its_reason(self) -> None: + rows = [clone_row("bash", "a/u.sh", "b/u.sh", 20)] + out = self.rendered(resummarized(duplication_doc(rows))) + self.assertIn( + "Excluded by a sanctioned-replication registry: 0 (no registry configured, or " + "none matched).", + out, + ) + doc = resummarized(duplication_doc(rows)) + doc["excluded"] = [{"registry": "r.txt", "line": 3, "path": "u.sh"}] + self.assertIn( + "Excluded by a sanctioned-replication registry: 1.", self.rendered(doc) + ) + + def test_no_detector_prints_one_headline_with_the_hint(self) -> None: + hint = "jscpd: https://github.com/kucherenko/jscpd (npm install -g jscpd)" + doc = duplication_doc( + [], + status="empty", + run=[ + { + "lane": lane, + "measure": "duplication", + "collector": None, + "status": "unavailable", + "reason": "jscpd: not on PATH", + "hint": hint, + } + for lane in ("bash", "python") + ], + unavailable=["bash/duplication", "python/duplication"], + ) + out = self.rendered(doc) + self.assertEqual(out.count("No clone detector ran in any lane"), 1) + self.assertEqual(out.count("npm install -g jscpd"), 1) + self.assertIn("/code-metrics:setup", out) + self.assertEqual(out.count("| unavailable | jscpd: not on PATH |"), 2) + self.assertLess(out.index("No clone detector"), out.index("## Coverage")) + + def test_a_lane_that_skipped_every_file_is_partial_not_empty(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + d = Path(tmp) + result = run( + "assemble", + "--skill", + "audit-duplication", + "--scope", + write(d, "s.json", json.dumps({"mode": "all", "files": 1})), + "--run", + write( + d, + "r.jsonl", + jsonl( + [ + { + "lane": "bash", + "measure": "duplication", + "collector": "jscpd 5.2.0", + "status": "partial", + "reason": "1 of 1 files skipped by duplication.max_size 1mb / max_lines none", + "hint": None, + } + ] + ), + ), + "--measures", + write(d, "m.jsonl", ""), + "--thresholds", + write(d, "t.json", "[]"), + ) + self.assertEqual(result.returncode, 0, result.stderr) + doc = json.loads(result.stdout) + self.assertEqual(doc["status"], "partial") + out = self.rendered(doc) + self.assertNotIn("Measured nothing", out) + self.assertIn("Status: partial", out) + self.assertIn("\nPartial: bash/duplication.\n", out) + + def test_a_size_document_renders_as_before(self) -> None: + doc = { + "schema": "code-metrics/v1", + "skill": "audit-size", + "status": "partial", + "scope": {"mode": "all", "base": None, "files": 1, "excluded": 0}, + "run": [ + { + "lane": "python", + "measure": "size", + "collector": "scc 3.4.0", + "status": "partial", + "reason": "1 of 2 files unreadable", + "hint": None, + } + ], + "thresholds": [], + "measures": [ + { + "file": "a.py", + "function": None, + "lane": "python", + "values": {"lines_non_blank": 12}, + "over_reference": ["file_lines"], + } + ], + "summary": { + "files": 1, + "functions": 0, + "over_reference": {"file_lines": 1}, + }, + "excluded": [], + "unavailable": [], + } + out = self.rendered(doc) + self.assertEqual( + out, + "# code-metrics: audit-size\n" + "\n" + "Status: partial. Scope: all, 1 file(s).\n" + "\n" + "## Coverage of this run\n" + "\n" + "| Lane | Measure | Collector | Status | Reason |\n" + "|---|---|---|---|---|\n" + "| python | size | scc 3.4.0 | partial | 1 of 2 files unreadable |\n" + "\n" + "## Measures\n" + "\n" + "| File | Function | Lane | lines_non_blank | Over reference |\n" + "|---|---|---|---|---|\n" + "| a.py | | python | 12 | file_lines |\n" + "\n" + "## Summary\n" + "\n" + "Files: 1. Functions: 0. Over reference: file_lines 1.\n", + ) + + if __name__ == "__main__": unittest.main() diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh b/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh index 6753f11343..4ecaabb6c5 100755 --- a/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh +++ b/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh @@ -84,7 +84,7 @@ if [[ -z "$CONFIG" ]]; then --home "${CODE_METRICS_HOME:-${HOME:-/}}" >"$CONFIG" || exit 2 fi -# Five tunables and then one line per configured registry, in that order. A +# Six tunables and then one line per configured registry, in that order. A # cap of null or 0 is exported empty, which the adapter reads as "no cap". mapfile -t DUP < <("${PY[@]}" -c ' import json, sys @@ -113,10 +113,11 @@ ignore = section.get("ignore") print(",".join(str(item) for item in ignore) if isinstance(ignore, list) else "") print(cap("max_lines")) print(cap("max_size")) +print(number("rollup_depth", 2)) for registry in section.get("registries") or []: print(str(registry)) ' "$CONFIG") -if [[ ${#DUP[@]} -lt 5 ]]; then +if [[ ${#DUP[@]} -lt 6 ]]; then echo "audit-duplication.sh: the resolved configuration could not be read" >&2 exit 2 fi @@ -125,6 +126,7 @@ export CODE_METRICS_DUP_MIN_LINES="${DUP[1]}" export CODE_METRICS_DUP_IGNORE="${DUP[2]}" export CODE_METRICS_DUP_MAX_LINES="${DUP[3]}" export CODE_METRICS_DUP_MAX_SIZE="${DUP[4]}" +ROLLUP_DEPTH="${DUP[5]}" FILTER_ARGS=(--root "$ROOT") resolve_registry() { @@ -137,7 +139,7 @@ resolve_registry() { return 1 fi } -for registry in "${REGISTRY_ARGS[@]:-}" "${DUP[@]:5}"; do +for registry in "${REGISTRY_ARGS[@]:-}" "${DUP[@]:6}"; do [[ -n "$registry" ]] || continue if ! resolved="$(resolve_registry "$registry")"; then echo "audit-duplication.sh: registry not found: $registry" >&2 @@ -155,12 +157,12 @@ rc=$? # recomputation drops when every group was excluded. "${PY[@]}" "$CLUSTER" <"$WORK/report.json" >"$WORK/clustered.json" || exit 2 "${PY[@]}" "$FILTER" "${FILTER_ARGS[@]}" <"$WORK/clustered.json" >"$WORK/filtered.json" || exit 2 -"${PY[@]}" "$REPORT" resummarize <"$WORK/filtered.json" >"$WORK/summed.json" || exit 2 +"${PY[@]}" "$REPORT" resummarize --root "$ROOT" <"$WORK/filtered.json" >"$WORK/summed.json" || exit 2 "${PY[@]}" "$FILTER" --zero-floor --root "$ROOT" <"$WORK/summed.json" >"$WORK/final.json" || exit 2 if [[ $JSON -eq 1 ]]; then cat "$WORK/final.json" else - "${PY[@]}" "$REPORT" render <"$WORK/final.json" || exit 2 + "${PY[@]}" "$REPORT" render --rollup-depth "$ROLLUP_DEPTH" <"$WORK/final.json" || exit 2 fi exit "$rc" diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py b/plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py index bc6fab2165..683264e6bc 100755 --- a/plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py +++ b/plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py @@ -196,6 +196,10 @@ def floor_summary(document: dict[str, Any]) -> dict[str, Any]: summary = document.setdefault("summary", {}) summary.setdefault("duplicated_lines", 0) summary.setdefault("clone_groups", 0) + # The rollups are derived from surviving groups the same way, so a + # clone-free run states them as empty maps rather than omitting them. + summary.setdefault("by_lane", {}) + summary.setdefault("by_directory", {}) return document diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py b/plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py index 9738f206ad..cc1025ce69 100755 --- a/plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py +++ b/plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py @@ -264,6 +264,14 @@ def test_the_zero_floor_states_zero_once_a_collector_ran(self) -> None: summary = json.loads(result.stdout)["summary"] self.assertEqual((summary["duplicated_lines"], summary["clone_groups"]), (0, 0)) + def test_the_zero_floor_states_empty_rollup_maps(self) -> None: + doc = document() + doc["run"] = [{"lane": "bash", "measure": "duplication", "status": "ok"}] + doc["summary"] = {"files": 0, "functions": 0, "over_reference": {}} + result = run(doc, "--root", ".", "--zero-floor") + summary = json.loads(result.stdout)["summary"] + self.assertEqual((summary["by_lane"], summary["by_directory"]), ({}, {})) + def test_the_zero_floor_counts_a_partial_lane_as_measured(self) -> None: doc = document() doc["run"] = [ From b7328c0ad1742e1dc0f99386406e2614557918cd Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 16:42:25 +0000 Subject: [PATCH 09/15] docs(code-metrics): describe clone classes, caps, cluster lines; release 0.1.9 Phase 5 of the duplication audit plan. - audit-duplication SKILL.md: the three new keys and the 0-means-null rule, both registry line shapes, the five exported tunables and why the jscpd adapter applies the caps itself, the no-detector headline and the offer-never-perform install rule, clone classes and the identity-not-overlap merge, the rollups and their root identity, the partial reading, and the gotchas for drifted copies and cross-major comparison. - README row and known-gaps list; CHANGELOG [0.1.9]; plugin.json 0.1.9. - The two-copy jscpd capture is regenerated under jscpd 5.2.0 with repo-relative names so no stale version string remains; the plan records the whole-tree dogfood numbers under both majors. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01QShQwS5yDYYToBmofdEY5N --- .../code-metrics-duplication-audit/PLAN.md | 16 ++- .../code-metrics/.claude-plugin/plugin.json | 2 +- plugins/code-metrics/CHANGELOG.md | 48 +++++++ plugins/code-metrics/README.md | 5 +- plugins/code-metrics/reference/collectors.md | 2 +- .../scripts/collectors/test_jscpd.py | 9 +- .../scripts/fixtures/tool-output/jscpd.json | 131 +++++++++--------- .../skills/audit-duplication/SKILL.md | 82 ++++++++--- 8 files changed, 198 insertions(+), 97 deletions(-) diff --git a/docs/topics/code-metrics-duplication-audit/PLAN.md b/docs/topics/code-metrics-duplication-audit/PLAN.md index 0965a7d8f3..17c7ba54a4 100644 --- a/docs/topics/code-metrics-duplication-audit/PLAN.md +++ b/docs/topics/code-metrics-duplication-audit/PLAN.md @@ -478,7 +478,7 @@ Review: code-design the install hint appears in that headline and again in each lane row's reason, which the dispatcher builds with the hint embedded and this phase leaves unchanged. -### Phase 5: SKILL.md, README, CHANGELOG, version, dogfood [TODO] +### Phase 5: SKILL.md, README, CHANGELOG, version, dogfood [DONE] 1. `skills/audit-duplication/SKILL.md`: configuration section names the three new keys, the `0`-means-null rule, and both registry line shapes; "Run it" gains the no-detector instruction @@ -517,6 +517,20 @@ Review: code-design `grep -c '5\.1\.2' plugins/code-metrics/reference/collectors.md plugins/code-metrics/scripts/collectors/test_jscpd.py plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh` prints `0` for each. - `markdownlint-cli2` over the changed markdown exits 0. +**Dogfood (whole tree, `--all --registry scripts/cross-plugin-source-registry.txt`, 2026-09-11):** + +| Detector | Status | Surviving classes | Duplicated lines | Files with clones | Excluded | Partial lanes | +|---|---|---|---|---|---|---| +| jscpd 5.2.0 | partial | 742 | 11831 | 487 | 14 (the five cluster lines: 18, 7, 4, 2, 2 instances) | typescript (1 of 295 files over 1mb: `plugins/miro/dist/index.min.js`) | +| jscpd 4.3.0 | partial | 625 | 10926 | 429 | 14 | typescript (same file) | + +Before this change the same 5.2.0 run reported 919 pair rows and 75267 duplicated lines with no +skipped file named. The two majors tokenize differently, so their counts are not comparable with +each other. The largest surviving classes are per-suite test-harness boilerplate (`pass`/`fail` +helpers shared by `lib/*.test.sh` and plugin test files), which no sync script declares and the +shell-test-helpers convention keeps per file: a finding for the operator, not sanctioned +replication. + ### Alternatives Considered | Alternative | Why rejected | Switch condition | diff --git a/plugins/code-metrics/.claude-plugin/plugin.json b/plugins/code-metrics/.claude-plugin/plugin.json index 2a5fb15132..14cf76ebc2 100644 --- a/plugins/code-metrics/.claude-plugin/plugin.json +++ b/plugins/code-metrics/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "code-metrics", - "version": "0.1.8", + "version": "0.1.9", "description": "Read-only code measures for a change, with cited references and no verdict: lines per file (audit-size), cyclomatic, cognitive, and Halstead complexity (audit-complexity), duplication with sanctioned-replication exclusions (audit-duplication), coverage per function with CRAP from existing lcov, Cobertura, coverage.py, or Go artifacts (audit-coverage), type debt for TypeScript and Python (audit-type-debt), the literacy router for what each number can and cannot say (principles), and a setup skill for the consumer's .claude/code-metrics.yaml. Runs external collectors only when they already resolve, never installs, never runs tests, never emits a finding.", "author": { "name": "Melodic Software", diff --git a/plugins/code-metrics/CHANGELOG.md b/plugins/code-metrics/CHANGELOG.md index a89098c149..1d3eabd5ac 100644 --- a/plugins/code-metrics/CHANGELOG.md +++ b/plugins/code-metrics/CHANGELOG.md @@ -3,6 +3,54 @@ All notable changes to the `code-metrics` plugin are documented here. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this plugin uses semantic versioning. +## [0.1.9] + +### Added + +- **`audit-duplication` merges detector pairs into clone classes.** `jscpd` and PMD CPD report a + clone as a pair, so N copies of one fragment arrived as N-1 rows and the summary counted the + fragment's lines N-1 times. A post-pass (`cluster-clones.py`) now joins rows that share an + instance with an identical file and line range into one row per class, the instances sorted by + path and the row labelled `clustered`; the lines count once. The merge joins on identity, not + overlap: a copy that shares only part of a fragment stays its own group. +- **Explicit size and line caps, reported instead of hidden.** `duplication.max_size` (default + `1mb`, binary units) and `duplication.max_lines` (default `null`) are applied by the jscpd + adapter before the tool runs, because jscpd 4 and 5 disagree on their own `--max-size` and + `--max-lines` defaults and on what `0` means, and neither names a skipped file. A skipped file + makes the lane's run row `partial` with the count and the largest file, the document `partial`, + and the markdown summary carries a `Partial:` line. `0` or `null` means no cap. +- **Registry cluster lines.** A sanctioned-replication registry line ` -> ...` + names a root-relative canonical copy and the plugin paths or gitignore-style globs that carry + it, so a canonical file outside any plugin (this repository's `lib/hook-utils.sh`) can declare + its copies; instance paths are compared root-relative and the first matching line wins. A plain + line is still one path-within-plugin taken whole. +- **Per-lane and per-directory rollups.** `summary.by_lane` and `summary.by_directory` (every + ancestor of each class's first instance, cumulative) are additive `code-metrics/v1` fields, + computed after registry exclusion; `duplication.rollup_depth` (default 2) decides how deep the + markdown `## Rollup` section lists. The schema reference states that readers ignore unknown keys. +- **Run rows carry the install hint as a field.** `run[].hint` holds the first install hint a + failed probe produced, apart from the prose reason, so a renderer can print it once. + +### Changed + +- **The duplication markdown reads as a duplication report.** Clone rows are listed largest + first; the summary line is `Files with clones: N.` instead of the size-shaped `Files. Functions. + Over reference.`; an empty exclusion list is stated with its reason; and a run in which no clone + detector resolved for any lane opens with one headline carrying the install hint and + `/code-metrics:setup`. The skill offers that install to the user and never performs it + unprompted. Every other skill's document renders as before. +- **`reference/collectors.md` pins jscpd 5.2.0** and records the 4.x maintenance line (4.3.0), + which the adapter also translates, the binary size grammar, and the token-count difference + between the majors. + +### Fixed + +- **A lane that skipped every file is `partial`, not `empty`**, and the zero floor counts a + `partial` duplication row as measured, so an all-excluded or clone-free lane that skipped a file + still states `duplicated_lines: 0`. +- **A run from a subdirectory matches the same registry lines as a run from the root**, because + instance paths are normalized against the repository root before matching. + ## [0.1.8] ### Added diff --git a/plugins/code-metrics/README.md b/plugins/code-metrics/README.md index 824d523d34..4fb381dda4 100644 --- a/plugins/code-metrics/README.md +++ b/plugins/code-metrics/README.md @@ -14,7 +14,7 @@ value to count against, not a bar. |---|---| | `/code-metrics:audit-complexity` | Per-function cyclomatic and cognitive complexity and Halstead difficulty from whichever collector resolves (`lizard`, `radon`, ESLint rules, `gocyclo`, `gocognit`, `shellmetrics`, `multimetric`), beside the ISO/IEC 5055 §8.2.117 reference of 20 with 10 and 15 selectable; cognitive and Halstead carry no standard threshold. | | `/code-metrics:audit-size` | Lines per file (total, blank, comment, code through `scc`; total and non-blank from a bundled counter otherwise) beside a cited reference; `size.mode: iso-8.2.115` adds the ISO function-percentage form. | -| `/code-metrics:audit-duplication` | Clone groups (duplicated lines and tokens, every instance's range) from `jscpd`, `dupl`, or PMD CPD, minus the replication the repository declares in a sanctioned-replication registry, which is an exclusion, not a suppression. | +| `/code-metrics:audit-duplication` | Clone classes (the detector's pairs merged; duplicated lines and tokens, every instance's range) from `jscpd`, `dupl`, or PMD CPD, rolled up per lane and per directory, minus the replication the repository declares in a sanctioned-replication registry (a path-within-plugin, or a `canonical -> copies` cluster line), which is an exclusion, not a suppression. A file over the size cap is reported as skipped, never silently dropped. | | `/code-metrics:audit-coverage` | Line coverage per file and per function read from the artifacts a build already produced (lcov 1.x and 2.2, Cobertura, coverage.py JSON, Go cover profile), plus CRAP per function from the complexity rows; it never runs a test, a missing artifact is a visible warning, and a function with no executable lines reports `null`, never zero. | | `/code-metrics:audit-type-debt` | The typed-code percentage per lane: `type-coverage` for TypeScript, mypy's `--any-exprs-report` for Python; no standard or CWE anchors the measure, so the reference is `null` by design. C# is reported as not applicable. | | `/code-metrics:principles` | Metric literacy: what each measure can and cannot tell you, where every reference value came from, CRAP's corrected provenance, the cross-metric caveats (carried once, here), and gated pointers to the plugins that own mutation score, tautological tests, dead code, coupling, and lint. | @@ -118,7 +118,8 @@ figure for a live session. `scripts/config-defaults.json`. The setup template and the `reference/config.md` key table both are, by a test and by `scripts/check-code-metrics-config-reference.py`; what remains unbound is the number written into a sentence or a small illustrative table, currently `coverage.reference` - in `audit-coverage`, `duplication.min_tokens` and `duplication.min_lines` in `audit-duplication`, + in `audit-coverage`, `duplication.min_tokens`, `duplication.min_lines`, `duplication.max_size`, + `duplication.max_lines`, and `duplication.rollup_depth` in `audit-duplication`, `type_debt.reference` in `audit-type-debt`, and the cyclomatic reference in `setup`. Those drift silently until someone reads them. diff --git a/plugins/code-metrics/reference/collectors.md b/plugins/code-metrics/reference/collectors.md index 3ba6449153..3179cd76c3 100644 --- a/plugins/code-metrics/reference/collectors.md +++ b/plugins/code-metrics/reference/collectors.md @@ -33,7 +33,7 @@ built them could not run the tool; the first live run is that row's recheck trig | Tool or format | Lane(s) | Measure | Claim the adapter relies on | Basis | Verified | Recheck trigger | |---|---|---|---|---|---|---| -| `jscpd` 5.2.0 | every lane | `duplication` | v5 is a Rust binary that only writes `/jscpd-report.json`, never stdout, so the adapter runs `--reporters json --output ` and prints the file itself; `duplicates[]` carries `firstFile`/`secondFile` with `start`, `end`, plus `lines` and `tokens`, and 5.2.0 adds a per-duplicate `kind` (`exact`) the adapter does not read; `--absolute` is required because names are otherwise relative to the common ancestor of the inputs, which collapses two vendored copies sharing a basename; `statistics.total.sources` counts token sources, not files, so it is not a skipped-file signal | github.com/kucherenko/jscpd, run in this repository (the capture is `scripts/fixtures/tool-output/jscpd.json`, a 5.1.2 capture whose keys 5.2.0 reproduces) | 2026-09-11 | a jscpd major release, or a change to the report filename or the `duplicates[]` shape | +| `jscpd` 5.2.0 | every lane | `duplication` | v5 is a Rust binary that only writes `/jscpd-report.json`, never stdout, so the adapter runs `--reporters json --output ` and prints the file itself; `duplicates[]` carries `firstFile`/`secondFile` with `start`, `end`, plus `lines` and `tokens`, and 5.2.0 adds a per-duplicate `kind` (`exact`) the adapter does not read; `--absolute` is required because names are otherwise relative to the common ancestor of the inputs, which collapses two vendored copies sharing a basename; `statistics.total.sources` counts token sources, not files, so it is not a skipped-file signal | github.com/kucherenko/jscpd, run in this repository (the capture is `scripts/fixtures/tool-output/jscpd.json`, from 5.2.0) | 2026-09-11 | a jscpd major release, or a change to the report filename or the `duplicates[]` shape | | `jscpd` 4.3.0 (the 4.x maintenance line) | every lane | `duplication` | v4 is a Node program that writes the same `/jscpd-report.json` with the same `firstFile`/`secondFile`, `start`, `end`, `lines`, and `tokens` keys, so one adapter translates both majors; it tokenizes differently from v5, so a clone count can differ between the majors on the same input, and the plugin never compares counts across a major boundary | github.com/kucherenko/jscpd (`npm install jscpd@4`), run in this repository over the same cluster fixture | 2026-09-11 | a 4.x release that changes the report shape, or the 4.x line being retired upstream | | `jscpd` size and line caps (both majors) | every lane | `duplication` | the adapter applies `duplication.max_size` and `duplication.max_lines` itself, before jscpd runs, and passes jscpd one above its own bound (or `1000000` lines and `1099511627776` bytes when there is no cap), because the majors disagree on the flags: 4.x defaults to 100kb and 1000 lines and reads `--max-lines 0` as that default, 5.x defaults to 1mb with no line cap, both read `--max-size 0` as skip every file, and neither names a skipped file in the report; jscpd's size grammar is binary (`1kb` is 1,024 bytes, `1mb` is 1,048,576) and the adapter uses the same multipliers | github.com/kucherenko/jscpd, both majors run in this repository with `--max-size` and `--max-lines` set to `0`, the default, and one byte or line below a fixture file's size | 2026-09-11 | either major changes a `--max-size`/`--max-lines` default or the meaning of `0`, or a report gains a skipped-file list | | PMD CPD 7.27.0 | typescript, python, go, dotnet | `duplication` | `pmd cpd --minimum-tokens N --format xml --language --file-list ` (one path per line) prints a namespaced `pmd-cpd` document whose `duplication` elements carry `lines` and `tokens` with one `file` child per instance (`path`, `line`, `endline`); CPD has no JSON reporter, no minimum-lines option, no ignore-glob option, and no Bash or shell language; exit 4 means duplications were found, not that the run failed | docs.pmd-code.org CPD user documentation, CLI reference, and report formats; the adapter and its fixture are unverified against a live run | 2026-09-05 | a PMD 8 release, a JSON reporter, a shell CPD language, or the first live run of this adapter | diff --git a/plugins/code-metrics/scripts/collectors/test_jscpd.py b/plugins/code-metrics/scripts/collectors/test_jscpd.py index a44e14d49d..ec8e3f9d50 100755 --- a/plugins/code-metrics/scripts/collectors/test_jscpd.py +++ b/plugins/code-metrics/scripts/collectors/test_jscpd.py @@ -6,11 +6,10 @@ copies the committed capture fixtures/tool-output/jscpd.json into the `--output` directory the adapter passes, the way jscpd 5 writes its own report (design T13; no executable is committed). The capture came from a live -jscpd 5.1.2 run over the two-copy cluster under -fixtures/sources/cluster/{alpha,beta}/shared/shared-utils.sh; 5.2.0 writes the -same document plus a per-duplicate `kind` the adapter does not read, and 4.3.0 -writes the same keys the adapter does read, so one capture stands in for both -majors and the stub only varies the version line. +jscpd 5.2.0 run over the two-copy cluster under +fixtures/sources/cluster/{alpha,beta}/shared/shared-utils.sh, rewritten to +repo-relative names; 4.3.0 writes the same keys the adapter reads, so one +capture stands in for both majors and the stub only varies the version line. """ from __future__ import annotations diff --git a/plugins/code-metrics/scripts/fixtures/tool-output/jscpd.json b/plugins/code-metrics/scripts/fixtures/tool-output/jscpd.json index 3cc07f78b3..f3171a6dbb 100644 --- a/plugins/code-metrics/scripts/fixtures/tool-output/jscpd.json +++ b/plugins/code-metrics/scripts/fixtures/tool-output/jscpd.json @@ -1,70 +1,71 @@ { - "duplicates": [ - { - "firstFile": { - "end": 41, - "endLoc": { - "column": 1, - "line": 41, - "position": 1002 - }, - "name": "plugins/code-metrics/scripts/fixtures/sources/cluster/alpha/shared/shared-utils.sh", - "start": 1, - "startLoc": { - "column": 0, - "line": 1, - "position": 0 - } - }, - "format": "bash", - "fragment": "# shellcheck shell=bash\n# Fixture source for the code-metrics duplication suites: a helper vendored\n# byte-identical into two sibling plugin directories, standing in for a\n# repository that deliberately replicates one path across its plugins. Never\n# executed, so it carries no shebang and no exec bit; kept lint-clean on\n# purpose. The copy under the sibling directory is byte-for-byte this file.\n\nHARVEST_LABEL=\"harvest\"\n\nannounce_start() {\n local subject=\"$1\"\n printf 'start %s %s\\n' \"$HARVEST_LABEL\" \"$subject\"\n}\n\nannounce_finish() {\n local subject=\"$1\"\n local outcome=\"${2:-unknown}\"\n printf 'finish %s %s %s\\n' \"$HARVEST_LABEL\" \"$subject\" \"$outcome\"\n}\n\ncollect_orchard() {\n local basket=\"$1\"\n shift\n local apple\n for apple in \"$@\"; do\n if [[ -z \"$apple\" ]]; then\n continue\n fi\n printf '%s/%s\\n' \"$basket\" \"$apple\"\n done\n}\n\nmeasure_basket() {\n local basket=\"$1\"\n if [[ -d \"$basket\" ]]; then\n find \"$basket\" -type f | wc -l\n return 0\n fi\n printf '0\\n'\n return 1\n}", - "isNew": false, - "lines": 41, - "secondFile": { - "end": 41, - "endLoc": { - "column": 1, - "line": 41, - "position": 1002 - }, - "name": "plugins/code-metrics/scripts/fixtures/sources/cluster/beta/shared/shared-utils.sh", - "start": 1, - "startLoc": { - "column": 0, - "line": 1, - "position": 0 - } - }, - "tokens": 110 + "duplicates": [ + { + "firstFile": { + "end": 41, + "endLoc": { + "column": 1, + "line": 41, + "position": 1002 + }, + "name": "plugins/code-metrics/scripts/fixtures/sources/cluster/alpha/shared/shared-utils.sh", + "start": 1, + "startLoc": { + "column": 0, + "line": 1, + "position": 0 } - ], - "statistics": { - "detectionDate": "2026-09-05T16:06:08.332Z", - "formats": { - "bash": { - "clones": 1, - "duplicatedLines": 40, - "duplicatedTokens": 110, - "lines": 82, - "newClones": 0, - "newDuplicatedLines": 0, - "percentage": 48.78048780487805, - "percentageTokens": 50.0, - "sources": 2, - "tokens": 220 - } + }, + "format": "bash", + "fragment": "# shellcheck shell=bash\n# Fixture source for the code-metrics duplication suites: a helper vendored\n# byte-identical into two sibling plugin directories, standing in for a\n# repository that deliberately replicates one path across its plugins. Never\n# executed, so it carries no shebang and no exec bit; kept lint-clean on\n# purpose. The copy under the sibling directory is byte-for-byte this file.\n\nHARVEST_LABEL=\"harvest\"\n\nannounce_start() {\n local subject=\"$1\"\n printf 'start %s %s\\n' \"$HARVEST_LABEL\" \"$subject\"\n}\n\nannounce_finish() {\n local subject=\"$1\"\n local outcome=\"${2:-unknown}\"\n printf 'finish %s %s %s\\n' \"$HARVEST_LABEL\" \"$subject\" \"$outcome\"\n}\n\ncollect_orchard() {\n local basket=\"$1\"\n shift\n local apple\n for apple in \"$@\"; do\n if [[ -z \"$apple\" ]]; then\n continue\n fi\n printf '%s/%s\\n' \"$basket\" \"$apple\"\n done\n}\n\nmeasure_basket() {\n local basket=\"$1\"\n if [[ -d \"$basket\" ]]; then\n find \"$basket\" -type f | wc -l\n return 0\n fi\n printf '0\\n'\n return 1\n}", + "isNew": false, + "kind": "exact", + "lines": 41, + "secondFile": { + "end": 41, + "endLoc": { + "column": 1, + "line": 41, + "position": 1002 }, - "total": { - "clones": 1, - "duplicatedLines": 40, - "duplicatedTokens": 110, - "lines": 82, - "newClones": 0, - "newDuplicatedLines": 0, - "percentage": 48.78048780487805, - "percentageTokens": 50.0, - "sources": 2, - "tokens": 220 + "name": "plugins/code-metrics/scripts/fixtures/sources/cluster/beta/shared/shared-utils.sh", + "start": 1, + "startLoc": { + "column": 0, + "line": 1, + "position": 0 } + }, + "tokens": 110 + } + ], + "statistics": { + "detectionDate": "2026-09-11T16:41:08.069Z", + "formats": { + "bash": { + "clones": 1, + "duplicatedLines": 40, + "duplicatedTokens": 110, + "lines": 82, + "newClones": 0, + "newDuplicatedLines": 0, + "percentage": 48.78048780487805, + "percentageTokens": 50.0, + "sources": 2, + "tokens": 220 + } + }, + "total": { + "clones": 1, + "duplicatedLines": 40, + "duplicatedTokens": 110, + "lines": 82, + "newClones": 0, + "newDuplicatedLines": 0, + "percentage": 48.78048780487805, + "percentageTokens": 50.0, + "sources": 2, + "tokens": 220 } -} + } +} \ No newline at end of file diff --git a/plugins/code-metrics/skills/audit-duplication/SKILL.md b/plugins/code-metrics/skills/audit-duplication/SKILL.md index 3d20e5ac81..fe6a4c0b94 100644 --- a/plugins/code-metrics/skills/audit-duplication/SKILL.md +++ b/plugins/code-metrics/skills/audit-duplication/SKILL.md @@ -1,5 +1,5 @@ --- -description: "Measure duplicated code as clone groups over the changed files, a path, or the whole tree: each group's duplicated lines and tokens with every instance's file and line range, per lane (TypeScript/JavaScript, Python, Bash, Go, C#) from whichever clone detector already resolves. Replication the target repository declares about itself, a file vendored into several plugins and listed by path-within-plugin in a sanctioned-replication registry, is subtracted from the total and reported as an exclusion naming the registry line rather than as debt, and the report emits no finding, no severity, and no exit-code gate. Use when: 'is this duplicated', 'find copy-paste code', 'clone detection', 'duplication report', 'how much of this change is copied', 'DRY check', 'redundant code', 'duplicated lines in the diff'; for lines per file use /code-metrics:audit-size, and for what a duplication number can and cannot support use /code-metrics:principles." +description: "Measure duplicated code as clone groups over the changed files, a path, or the whole tree: each group's duplicated lines and tokens with every instance's file and line range, per lane (TypeScript/JavaScript, Python, Bash, Go, C#) from whichever clone detector already resolves. Detector pairs are merged into clone classes so copies count once, and the report rolls classes up per lane and per directory. Replication the target repository declares about itself, a file vendored into several plugins and listed in a sanctioned-replication registry by path-within-plugin or as a canonical-to-copies cluster line, is subtracted from the total and reported as an exclusion naming the registry line rather than as debt, and the report emits no finding, no severity, and no exit-code gate. Use when: 'is this duplicated', 'find copy-paste code', 'clone detection', 'duplication report', 'how much of this change is copied', 'DRY check', 'redundant code', 'duplicated lines in the diff'; for lines per file use /code-metrics:audit-size, and for what a duplication number can and cannot support use /code-metrics:principles." argument-hint: "[--json] [--all] [--base ] [--registry ] [...]" user-invocable: true disable-model-invocation: false @@ -44,9 +44,13 @@ continues. This plugin never installs, downloads, or `npx`-fetches a detector. ``` Present the markdown report as printed. It opens with the scope and a "Coverage of this run" -table (lane, collector, status, reason), then one row per clone group listing every instance as -`file:start-end`, then the summary line with the duplicated-line total and how many groups a -registry excluded. Keep the `--json` document when the numbers feed a comparison: +table (lane, collector, status, reason), then one row per clone group, largest first, listing +every instance as `file:start-end`, then a rollup per lane and per directory, then the summary +lines: files with clones, the duplicated-line total, how many groups a registry excluded, and +which lanes were partial. When the report opens with `No clone detector ran in any lane`, offer +the user the install command that headline carries (`npm install -g jscpd`, or a devDependency) +and run it only when they confirm; never install silently and never `npx`-fetch it. Keep the +`--json` document when the numbers feed a comparison: `/verification:measure metrics` consumes it when the `verification` plugin is installed (treat a report whose `status` is `empty` on either side as INCONCLUSIVE); otherwise keep the JSON beside your notes and compare by hand. @@ -55,9 +59,24 @@ your notes and compare by hand. - A clone group is reported beside no reference. There is no configured bar for duplication in this plugin and no standard sets one, so nothing is ever counted as `over_reference`. -- `summary.duplicated_lines` counts each group once, using the length of the group, not the sum - over its instances: two copies of a 41-line block are 41 duplicated lines, not 82. The count is - what survived the registries. +- A group is a clone class, not a detector pair. `jscpd` and PMD CPD report clones as pairs, so + seventeen identical copies arrive as sixteen two-instance rows; this skill merges rows that + share an instance with an identical file and line range into one row per class before it + counts anything. Copies that share only part of a fragment are named with different ranges and + stay separate groups: the merge joins on identity, never on overlap, so a class is never wider + than what the detector called identical. +- `summary.duplicated_lines` counts each class once, using the length of the class, not the sum + over its instances: three copies of a 41-line block are 41 duplicated lines, not 82 or 123. The + count is what survived the registries. +- `summary.by_lane` and `summary.by_directory` roll the surviving classes up: each maps to + `{groups, duplicated_lines}`, the directory map for `.` and every ancestor of each class's first + instance. A class counts once under every ancestor, so a parent includes its children and the + directory rows cannot be summed; `by_directory["."]` and the per-lane sum both restate the + totals. The markdown lists directories to `duplication.rollup_depth`; the JSON carries all. +- A file larger than `duplication.max_size` (or longer than `duplication.max_lines`, when set) is + left out of the scan and never silently dropped: the lane's run row is `partial` with how many + files were skipped and the largest one, the document is `partial`, and the summary carries a + `Partial:` line naming the lane. - The registry is an **exclusion**, not a suppression: it is derived from the target repository's own declaration that those copies are deliberate, so no suppression record is involved and the excluded groups stay in the document under `excluded[]` with the registry path, the 1-based line @@ -66,9 +85,10 @@ your notes and compare by hand. directory are ordinary duplication and stay. - A value the detector did not produce is `null`, never `0`: `dupl` reports no token count, so its rows carry `tokens: null`. -- `status` is `complete` when every lane in scope was measured, `partial` when one was not, and - `empty` when nothing was; a run that measured nothing prints "Measured nothing" and states no - duplication figure at all, which is not the same as zero duplication. +- `status` is `complete` when every lane in scope was measured, `partial` when one was not or + when a cap left files out of one, and `empty` when nothing was; a run that measured nothing + prints "Measured nothing" and states no duplication figure at all, which is not the same as zero + duplication. A lane that skipped every file is still `partial`: its row says what was skipped. - Exit 0 whenever a report was produced, including an `empty` one; exit 2 for a usage error such as a named registry or scope path that does not exist; exit 3 when a detector resolved but produced nothing parseable, with its stderr in the run table. A detector's own non-zero exit is @@ -79,16 +99,31 @@ your notes and compare by hand. Everything tunable resolves through `.claude/code-metrics.yaml` (user-global, team, local overlay; per-key override; keys in `${CLAUDE_PLUGIN_ROOT}/reference/config.md`): `duplication.min_tokens` (default 50), `duplication.min_lines` (default 5), -`duplication.ignore` (globs handed to the detector's own ignore option), and -`duplication.registries` (sanctioned-replication registries, each path relative to the repository -root, and each also nameable on the command line with `--registry`). `/code-metrics:setup` writes -the team file and probes the collectors. - -This script exports the three tunables to the collector adapters as -`CODE_METRICS_DUP_MIN_TOKENS`, `CODE_METRICS_DUP_MIN_LINES`, and `CODE_METRICS_DUP_IGNORE`, which -is the only channel an adapter reads them through. `jscpd` passes all three to the tool; -`dupl` and `cpd` have no minimum-lines or ignore-glob option, so their adapters apply the -minimum after parsing and report the ignore globs as unused. +`duplication.ignore` (globs handed to the detector's own ignore option), `duplication.max_size` +(default `1mb`, binary units; a larger file is left out of the scan and reported), `duplication.max_lines` +(default `null`, no line cap), `duplication.rollup_depth` (default 2, how deep the markdown +per-directory rollup lists), and `duplication.registries` (sanctioned-replication registries, each +path relative to the repository root, and each also nameable on the command line with +`--registry`). A cap of `null` or `0` means no cap. `/code-metrics:setup` writes the team file +and probes the collectors. + +A registry line has one of two shapes. A plain line is one path-within-plugin, taken whole with +any spaces: a class is excluded when every instance ends with that path and the copies sit in +distinct carrying directories. A cluster line, ` -> ...`, names a root-relative +canonical copy and the plugin paths or gitignore-style globs that carry it +(`lib/hook-utils.sh -> plugins/*/hooks/hook-utils.sh`): a class is excluded when every instance is +the canonical or matches a member and the instances' directories are pairwise distinct. Instance +paths are compared root-relative, so a run from a subdirectory matches the same lines, and the +first matching line in file order wins. + +This script exports the five tunables to the collector adapters as +`CODE_METRICS_DUP_MIN_TOKENS`, `CODE_METRICS_DUP_MIN_LINES`, `CODE_METRICS_DUP_IGNORE`, +`CODE_METRICS_DUP_MAX_LINES`, and `CODE_METRICS_DUP_MAX_SIZE`, which is the only channel an +adapter reads them through. `jscpd` passes the first three to the tool and applies the two caps +itself before the tool runs, because jscpd 4 and 5 disagree on what their own `--max-size` and +`--max-lines` default to and neither names a file it skipped; `dupl` and `cpd` have no +minimum-lines, ignore-glob, or cap option, so their adapters apply the minimum after parsing, +report the ignore globs as unused, and scan every file in scope. `cpd` (PMD) sits after `jscpd` on `${CLAUDE_PLUGIN_ROOT}/scripts/collector-ladder.tsv` for every lane but Bash, so it runs only when `jscpd` does not resolve and `pmd` does. A repository that @@ -123,7 +158,10 @@ overrides are validated against the ladder file and an unknown name is dropped w - Clone detection compares the files in scope with each other. A default-scope run sees only the changed files, so a block copied from a file the change did not touch is not found; use `--all` or name both paths when that is the question. -- `jscpd` reports pairs, so seventeen identical copies arrive as sixteen two-instance groups - rather than one seventeen-instance group; the registry excludes each of them on the same line. +- The pair merge is exact only for byte-identical copies. A class whose copies drifted by a line + is reported as the detector saw it: the identical span as one class, and the drifted copy's + shorter overlap as a second group naming the same file with a different range. +- jscpd 4 and jscpd 5 tokenize differently, so the same tree yields different class counts under + the two majors; compare runs made with one detector version, never across the boundary. - Lowering `duplication.min_tokens` finds more and smaller clones, most of them boilerplate the language forces; the defaults are the detector's own conservative pair. From 8455dc69be89f7701e5c6d20689703087216f1f1 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 16:57:33 +0000 Subject: [PATCH 10/15] docs(adr): record the registry cluster-line grammar as ADR 0033 The sanctioned-replication registry is a contract three readers parse, so the arrow line that declares a root canonical plus its copies is recorded with the alternatives it displaced. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01QShQwS5yDYYToBmofdEY5N --- ...l-plus-copies-cluster-as-one-arrow-line.md | 40 +++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 docs/adr/0033-declare-a-canonical-plus-copies-cluster-as-one-arrow-line.md diff --git a/docs/adr/0033-declare-a-canonical-plus-copies-cluster-as-one-arrow-line.md b/docs/adr/0033-declare-a-canonical-plus-copies-cluster-as-one-arrow-line.md new file mode 100644 index 0000000000..934c700f87 --- /dev/null +++ b/docs/adr/0033-declare-a-canonical-plus-copies-cluster-as-one-arrow-line.md @@ -0,0 +1,40 @@ +# Declare a canonical-plus-copies cluster as one arrow line + +- Status: accepted +- Date: 2026-09-11 + +## Context + +`scripts/cross-plugin-source-registry.txt` declares the replication this +repository does on purpose, one path-within-plugin per line, and three readers +key on that shape: the drift checker (`scripts/check-cross-plugin-source-drift.sh`), +the code-metrics replica collapser (`replica-collapse.py`), and the +duplication audit's exclusion filter (`registry-filter.py`). A path-within-plugin +cannot name a canonical copy that lives outside every plugin, so the eighteen +byte-identical copies of `hook-utils.sh` (root `lib/` plus seventeen plugins) +survived the duplication audit as one class with eighteen instances: the +seventeen plugin copies matched the line and the root copy did not. + +## Decision + +**A registry line containing ` -> ` is a cluster line:** the text before the +arrow is the root-relative canonical copy, the whitespace-separated tokens +after it are the members, each a literal root-relative path or a gitignore-style +glob (`lib/hook-utils.sh -> plugins/*/hooks/hook-utils.sh`). The duplication +filter excludes a clone class when every instance is the canonical or matches a +member and the instances sit in pairwise distinct directories; the drift +checker and the replica collapser skip the line, because they key clusters by +path-within-plugin and a root path is not one. A plain line keeps its meaning, +taken whole with any spaces. Lines are tried in file order and the first match +wins. + +## Why + +The registry is a contract every reader parses, so its grammar is hard to +change once lines exist. Splitting on whitespace was rejected: the drift +checker deliberately protects a registered path that contains a space, and a +second file or a YAML registry would have doubled the surface every reader +resolves. A marker that cannot occur in a path-within-plugin (` -> `) lets the +readers that do not understand a cluster ignore it with one test and lets the +one reader that does carry the whole class, canonical included, as a single +exclusion the report names by its line. From 6fd57df24652eee5143dd1464cad3ac958c5d20c Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 17:01:06 +0000 Subject: [PATCH 11/15] fix(code-metrics): pass jscpd a line bound no real file reaches Review finding: with no line cap configured the adapter passed jscpd --max-lines 1000000, a number a generated file can reach, so such a file would survive the adapter's pre-filter and then be skipped by jscpd's own gate with nothing named, the exact failure the pre-filter exists to close. The sentinel is now the signed 32-bit maximum, which both majors accept (verified against 4.3.0 and 5.2.0); tests and the collectors reference follow. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01QShQwS5yDYYToBmofdEY5N --- docs/topics/code-metrics-duplication-audit/PLAN.md | 6 ++++-- plugins/code-metrics/reference/collectors.md | 2 +- plugins/code-metrics/scripts/collectors/jscpd.py | 8 +++++--- plugins/code-metrics/scripts/collectors/test_jscpd.py | 4 ++-- .../audit-duplication/scripts/audit-duplication.test.sh | 2 +- 5 files changed, 13 insertions(+), 9 deletions(-) diff --git a/docs/topics/code-metrics-duplication-audit/PLAN.md b/docs/topics/code-metrics-duplication-audit/PLAN.md index e1d7e72c32..587630bc97 100644 --- a/docs/topics/code-metrics-duplication-audit/PLAN.md +++ b/docs/topics/code-metrics-duplication-audit/PLAN.md @@ -254,7 +254,9 @@ Review: code-design (`kb` = 1024, `mb` = 1,048,576, bare digits = bytes) and `CODE_METRICS_DUP_MAX_LINES`; treat an empty or `0` value as no cap; pre-filter the file list by `os.stat` size and, only when a line cap is set, by a binary-mode newline count; pass `--max-size ` always and - `--max-lines ` always, so the pre-filter is the only gate on both + `--max-lines ` always (the review raised the sentinel from + 1000000, which a generated file can reach, to the signed 32-bit maximum both majors accept), so + the pre-filter is the only gate on both majors [EXEC-SHAPE] (4.x reads `--max-lines 0` as "use the 1000 default" and both majors read a `0` size as "skip all"); when files were skipped, write one line to the partial-reason file (`N of M files skipped by duplication.max_size / max_lines ; largest: ()`), @@ -292,7 +294,7 @@ Review: code-design - `python3 scripts/check-code-metrics-config-reference.py` exits 0; `python3 -m unittest plugins/code-metrics/skills/setup/scripts/test_setup_apply.py` exits 0. - `python3 -m unittest plugins/code-metrics/scripts/collectors/test_jscpd.py` exits 0; its argv-log - case asserts `--max-size 1048577` and `--max-lines 1000000` present and no argument equal to `0` + case asserts `--max-size 1048577` and `--max-lines 2147483647` present and no argument equal to `0` follows either flag. - `bash plugins/code-metrics/scripts/dispatch.test.sh` exits 0 with a case whose JSON has a run row `status == "partial"` and a reason matching `^[0-9]+ of [0-9]+ files skipped`, and a case whose diff --git a/plugins/code-metrics/reference/collectors.md b/plugins/code-metrics/reference/collectors.md index 8ef0ca656c..4b24d9fd0c 100644 --- a/plugins/code-metrics/reference/collectors.md +++ b/plugins/code-metrics/reference/collectors.md @@ -35,7 +35,7 @@ built them could not run the tool; the first live run is that row's recheck trig |---|---|---|---|---|---|---| | `jscpd` 5.2.0 | every lane | `duplication` | v5 is a Rust binary that only writes `/jscpd-report.json`, never stdout, so the adapter runs `--reporters json --output ` and prints the file itself; `duplicates[]` carries `firstFile`/`secondFile` with `start`, `end`, plus `lines` and `tokens`, and 5.2.0 adds a per-duplicate `kind` (`exact`) the adapter does not read; `--absolute` is required because names are otherwise relative to the common ancestor of the inputs, which collapses two vendored copies sharing a basename; `statistics.total.sources` counts token sources, not files, so it is not a skipped-file signal | github.com/kucherenko/jscpd, run in this repository (the capture is `scripts/fixtures/tool-output/jscpd.json`, from 5.2.0) | 2026-09-11 | a jscpd major release, or a change to the report filename or the `duplicates[]` shape | | `jscpd` 4.3.0 (the 4.x maintenance line) | every lane | `duplication` | v4 is a Node program that writes the same `/jscpd-report.json` with the same `firstFile`/`secondFile`, `start`, `end`, `lines`, and `tokens` keys, so one adapter translates both majors; it tokenizes differently from v5, so a clone count can differ between the majors on the same input, and the plugin never compares counts across a major boundary | github.com/kucherenko/jscpd (`npm install jscpd@4`), run in this repository over the same cluster fixture | 2026-09-11 | a 4.x release that changes the report shape, or the 4.x line being retired upstream | -| `jscpd` size and line caps (both majors) | every lane | `duplication` | the adapter applies `duplication.max_size` and `duplication.max_lines` itself, before jscpd runs, and passes jscpd one above its own bound (or `1000000` lines and `1099511627776` bytes when there is no cap), because the majors disagree on the flags: 4.x defaults to 100kb and 1000 lines and reads `--max-lines 0` as that default, 5.x defaults to 1mb with no line cap, both read `--max-size 0` as skip every file, and neither names a skipped file in the report; jscpd's size grammar is binary (`1kb` is 1,024 bytes, `1mb` is 1,048,576) and the adapter uses the same multipliers | github.com/kucherenko/jscpd, both majors run in this repository with `--max-size` and `--max-lines` set to `0`, the default, and one byte or line below a fixture file's size | 2026-09-11 | either major changes a `--max-size`/`--max-lines` default or the meaning of `0`, or a report gains a skipped-file list | +| `jscpd` size and line caps (both majors) | every lane | `duplication` | the adapter applies `duplication.max_size` and `duplication.max_lines` itself, before jscpd runs, and passes jscpd one above its own bound (or `2147483647` lines and `1099511627776` bytes when there is no cap, bounds no real file reaches), because the majors disagree on the flags: 4.x defaults to 100kb and 1000 lines and reads `--max-lines 0` as that default, 5.x defaults to 1mb with no line cap, both read `--max-size 0` as skip every file, and neither names a skipped file in the report; jscpd's size grammar is binary (`1kb` is 1,024 bytes, `1mb` is 1,048,576) and the adapter uses the same multipliers | github.com/kucherenko/jscpd, both majors run in this repository with `--max-size` and `--max-lines` set to `0`, the default, and one byte or line below a fixture file's size | 2026-09-11 | either major changes a `--max-size`/`--max-lines` default or the meaning of `0`, or a report gains a skipped-file list | | PMD CPD 7.27.0 | typescript, python, go, dotnet | `duplication` | `pmd cpd --minimum-tokens N --format xml --language --file-list ` (one path per line) prints a namespaced `pmd-cpd` document whose `duplication` elements carry `lines` and `tokens` with one `file` child per instance (`path`, `line`, `endline`); CPD has no JSON reporter, no minimum-lines option, no ignore-glob option, and no Bash or shell language; exit 4 means duplications were found, not that the run failed | docs.pmd-code.org CPD user documentation, CLI reference, and report formats; the adapter and its fixture are unverified against a live run | 2026-09-05 | a PMD 8 release, a JSON reporter, a shell CPD language, or the first live run of this adapter | | `dupl` v1.1.0 | go | `duplication` | the default text printer emits `found clones:` per group, then an indented `:,` line per instance, then a total footer; `-plumbing` is pairwise and loses groups of three or more, so the text printer is parsed; `-t` is a token threshold with no line equivalent; dupl reports no token count and ships no version flag | github.com/mibk/dupl `printer/text.go` and `main.go`; the adapter and its fixture are unverified against a live run | 2026-09-05 | a dupl release that changes the printer, adds a version flag, or adds a token count | diff --git a/plugins/code-metrics/scripts/collectors/jscpd.py b/plugins/code-metrics/scripts/collectors/jscpd.py index b000a43221..2357219b44 100755 --- a/plugins/code-metrics/scripts/collectors/jscpd.py +++ b/plugins/code-metrics/scripts/collectors/jscpd.py @@ -69,9 +69,11 @@ DEFAULT_MIN_TOKENS = "50" DEFAULT_MIN_LINES = "5" DEFAULT_MAX_SIZE = "1mb" -# Passed to jscpd when the adapter applies no cap of its own: bounds no file -# that survives the pre-filter reaches, valid on both majors (`0` is not). -NO_LINE_CAP = 1_000_000 +# Passed to jscpd when the adapter applies no cap of its own: bounds no real +# file reaches, so jscpd's own gate can never skip a file this adapter did not +# name; both are accepted by both majors (`0` is not), verified 2026-09-11 +# against 4.3.0 and 5.2.0 with the line bound at the signed 32-bit maximum. +NO_LINE_CAP = 2_147_483_647 NO_SIZE_CAP = 1 << 40 _SIZE_UNITS = {"": 1, "b": 1, "kb": 1024, "mb": 1024**2, "gb": 1024**3} _SIZE_RE = re.compile(r"^(\d+(?:\.\d+)?)\s*([kmg]?b)?$") diff --git a/plugins/code-metrics/scripts/collectors/test_jscpd.py b/plugins/code-metrics/scripts/collectors/test_jscpd.py index ec8e3f9d50..238f53d1cb 100755 --- a/plugins/code-metrics/scripts/collectors/test_jscpd.py +++ b/plugins/code-metrics/scripts/collectors/test_jscpd.py @@ -228,7 +228,7 @@ def test_explicit_caps_reach_the_command_line_on_both_majors(self) -> None: # One byte above the adapter's own bound, so the pre-filter is # the only gate on either major. self.assertIn("--max-size 1048577", argv, version) - self.assertIn("--max-lines 1000000", argv, version) + self.assertIn("--max-lines 2147483647", argv, version) def test_a_zero_cap_means_no_cap_and_is_never_passed(self) -> None: with tempfile.TemporaryDirectory() as tmp: @@ -249,7 +249,7 @@ def test_a_zero_cap_means_no_cap_and_is_never_passed(self) -> None: argv = log.read_text(encoding="utf-8") self.assertNotIn("--max-size 0 ", argv + " ") self.assertNotIn("--max-lines 0 ", argv + " ") - self.assertIn("--max-lines 1000000", argv) + self.assertIn("--max-lines 2147483647", argv) self.assertIn("--max-size 1099511627776", argv) def test_the_size_grammar_uses_binary_multipliers(self) -> None: diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh b/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh index 1a7eecc34c..79ebd93a0f 100755 --- a/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh +++ b/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh @@ -159,7 +159,7 @@ assert_contains "min_tokens reaches the collector" "$argv" "--min-tokens 77" assert_contains "min_lines reaches the collector" "$argv" "--min-lines 9" assert_contains "the ignore globs reach the collector" "$argv" "--ignore **/vendor/**" assert_contains "max_size reaches the collector one byte above the bound" "$argv" "--max-size 8193" -assert_contains "a max_lines of 0 means no cap and reaches the collector as the explicit large value" "$argv" "--max-lines 1000000" +assert_contains "a max_lines of 0 means no cap and reaches the collector as the explicit large value" "$argv" "--max-lines 2147483647" # 7. Three byte-identical copies are one clone class, its lines counted once. # jscpd pairs each later copy with the first, so the capture holds two pairs From beb6169df0d4271c961bfcdd1f388a751181b4bc Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 17:47:15 +0000 Subject: [PATCH 12/15] fix(code-metrics): attribute a clone class the same way from any directory Verification finding: cluster-clones.py sorted a class's instances by the cwd-relative path the detector gave them, so a run from lib/ put `../plugins/...` ahead of `hook-utils.sh` and the directory rollup attributed the class elsewhere than a run from the root did. The merge now takes --root and sorts by the root-relative path (the instances keep their paths); the skill passes it. The root-relative helper the summarizer and the registry filter each carried moves into pathglob.py and all three share it. The plan's acceptance criteria are amended to the facts the verifier established: the two majors can name a byte-identical class with different line ranges (5.x drops a leading comment block), the typescript partial row is observable only with the repository's default dist exclusion lifted, and the version is the next patch above the default branch. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01QShQwS5yDYYToBmofdEY5N --- .../code-metrics-duplication-audit/PLAN.md | 35 +++++++++------ plugins/code-metrics/CHANGELOG.md | 4 +- plugins/code-metrics/scripts/pathglob.py | 22 ++++++++++ plugins/code-metrics/scripts/report.py | 19 ++------ plugins/code-metrics/scripts/test_pathglob.py | 26 +++++++++++ .../scripts/audit-duplication.sh | 2 +- .../scripts/cluster-clones.py | 38 +++++++++++----- .../scripts/registry-filter.py | 12 +----- .../scripts/test_cluster_clones.py | 43 ++++++++++++++++++- 9 files changed, 149 insertions(+), 52 deletions(-) diff --git a/docs/topics/code-metrics-duplication-audit/PLAN.md b/docs/topics/code-metrics-duplication-audit/PLAN.md index 587630bc97..333fb7dd0e 100644 --- a/docs/topics/code-metrics-duplication-audit/PLAN.md +++ b/docs/topics/code-metrics-duplication-audit/PLAN.md @@ -64,11 +64,16 @@ this repository the audit reads clean apart from genuine duplication. ### Acceptance criteria - Running `audit-duplication.sh --json --all` on this repository with jscpd 4.3.0 and again with - 5.2.0 yields, for every byte-identical whole-file class, the same set of instances (file and - line range) per group; `tokens` and instance order are excluded from the comparison because the - two majors tokenize differently and hub on different copies. The bash row reads `ok` under both; - the typescript row reads `partial` under both, naming `plugins/miro/dist/index.min.js` as the one - file over the 1mb cap. + 5.2.0 yields, for every byte-identical whole-file class, the same set of instance files per + group; `tokens`, instance order, and the exact line range are excluded from the comparison + because the two majors tokenize differently and hub on different copies (5.x drops a leading + comment block from the clone, so a file that opens with one is reported from its first code line + under 5.x and from line 1 under 4.x; the verifier found one such class, + `resolve-hook.mjs` in two `knowledge` skills). The bash row reads `ok` under both; with the + repository's default `**/dist/**` scope exclusion lifted, the typescript row reads `partial` + under both, naming `plugins/miro/dist/index.min.js` as the one file over the 1mb cap; with the + exclusion in force (this repository's shipped team file) the bundle never reaches the cap and the + row reads `ok`. - `duplication.max_lines` defaults to `null` (no cap) and `duplication.max_size` to `1mb`; both are documented in `reference/config.md` (gated against `config-defaults.json`) and exported to the adapter, and the adapter's tests cover the explicit-cap argv on both majors, the `0`-means-null @@ -121,9 +126,12 @@ this repository the audit reads clean apart from genuine duplication. - The jscpd adapter's docstring and `reference/collectors.md` state that 4.x and 5.x are both translated, pin 5.2.0, note the `kind` field, and record that the two majors tokenize differently; the schema reference names "intentional clones" beside "sanctioned replication". -- `plugin.json` reads 0.1.9 and `CHANGELOG.md` carries a `[0.1.9]` entry covering every item above; - `scripts/affected-tests.sh --run` exits 0, or exits 3 with only Python suites listed as not run, - each of which then passes under pytest. +- `plugin.json` carries the next patch version above the one on the default branch at merge time + (0.2.2 over 0.2.1; the Brief was drafted against 0.1.8) and `CHANGELOG.md` carries the matching + entry covering every item above; `scripts/affected-tests.sh --run` exits 0, or exits 3 with only + Python and Node suites listed as not run, each of which then passes in its own lane, or exits 1 + only for a suite that fails identically on a detached `origin/main` worktree and touches no file + this change edits (named in the PR body). ### Captured assumptions @@ -134,12 +142,15 @@ this repository the audit reads clean apart from genuine duplication. duplicated lines are totalled, and jscpd v5 is the one tool that sums per pair. Revisit if a standard sets a duplicated-lines definition. - The merge keys on identical (file, start_line, end_line) instances and equal `lines`, so only - byte-aligned copies join; jscpd extends a clone greedily into shared flanking lines, so offset - copies get ranges differing by a line and stay separate on both majors. Closure is exact only for + byte-aligned copies join; jscpd 5.x hubs every later copy on the first and names it with the same + range, so full copies at different offsets are one class, while 4.x hubs on the last input and + extends into shared flanking lines, so under 4.x the same copies can stay two groups (a live + probe of three offset copies gave one class under 5.2.0 and two groups under 4.3.0). Closure is exact only for type-1/type-2 clones, which is all the adapter receives because it passes no `--max-gap-lines`. Revisit if `similar` clones are ever enabled. -- Merged instances are sorted by path so the first instance, and therefore `by_directory` - attribution, is the same under 4.x (which hubs on the last input) and 5.x (which hubs on the first). +- Merged instances are sorted by root-relative path (`cluster-clones.py --root`) so the first + instance, and therefore `by_directory` attribution, is the same under 4.x (which hubs on the last + input) and 5.x (which hubs on the first), and the same whichever directory the run started from. - `rollup_depth` default 2 is the plugin's choice; no upstream sets a depth (SonarQube and Codacy roll up every directory). Revisit if a consuming repository's layout makes depth 2 meaningless. - The byte cap of 1mb aligns with jscpd 5.0.7's parser guard and SonarJS's 1000kb generated-code diff --git a/plugins/code-metrics/CHANGELOG.md b/plugins/code-metrics/CHANGELOG.md index 67fab80c0e..8d78ddd2f9 100644 --- a/plugins/code-metrics/CHANGELOG.md +++ b/plugins/code-metrics/CHANGELOG.md @@ -27,7 +27,9 @@ All notable changes to the `code-metrics` plugin are documented here. Format fol - **Per-lane and per-directory rollups.** `summary.by_lane` and `summary.by_directory` (every ancestor of each class's first instance, cumulative) are additive `code-metrics/v1` fields, computed after registry exclusion; `duplication.rollup_depth` (default 2) decides how deep the - markdown `## Rollup` section lists. The schema reference states that readers ignore unknown keys. + markdown `## Rollup` section lists. A class is attributed by its first instance after a + root-relative sort, so the rollup reads the same from the repository root and from a + subdirectory. The schema reference states that readers ignore unknown keys. - **Run rows carry the install hint as a field.** `run[].hint` holds the first install hint a failed probe produced, apart from the prose reason, so a renderer can print it once. diff --git a/plugins/code-metrics/scripts/pathglob.py b/plugins/code-metrics/scripts/pathglob.py index 47a1e91137..b62069024a 100755 --- a/plugins/code-metrics/scripts/pathglob.py +++ b/plugins/code-metrics/scripts/pathglob.py @@ -25,6 +25,7 @@ from __future__ import annotations +import os import re import sys @@ -40,6 +41,27 @@ def _normalize(path: str) -> str: return path +def root_relative(path: str, root: str) -> str: + """The path relative to `root` with forward slashes; unchanged without a root. + + A cwd-relative path is joined onto the working directory first, so a run + from a subdirectory (where the dispatcher names files `../../lib/x.sh`) and + a run from the root name a file the same way. Shared by the report + summarizer, the registry filter, and the clone-class merge so the three + never disagree on what "root-relative" means. + """ + path = (path or "").replace("\\", "/") + if root: + absolute = path if os.path.isabs(path) else os.path.join(os.getcwd(), path) + try: + path = os.path.relpath(absolute, root).replace("\\", "/") + except ValueError: + pass + while path.startswith("./"): + path = path[2:] + return path + + def translate(pattern: str) -> str: """Return an anchored regex for a gitignore-style glob.""" pattern = _normalize(pattern) diff --git a/plugins/code-metrics/scripts/report.py b/plugins/code-metrics/scripts/report.py index 50d52724f1..2142c55ebf 100755 --- a/plugins/code-metrics/scripts/report.py +++ b/plugins/code-metrics/scripts/report.py @@ -45,10 +45,11 @@ import argparse import datetime as _dt import json -import os import sys from typing import Any +from pathglob import root_relative + MIN_PYTHON = (3, 9) SCHEMA = "code-metrics/v1" RUN_STATUSES = ("ok", "partial", "unavailable", "not-applicable", "deferred") @@ -120,20 +121,6 @@ def _over(threshold: dict[str, Any], value: Any) -> bool: return value >= reference -def _root_relative(path: str, root: str) -> str: - """The path relative to `root` with forward slashes; unchanged without a root.""" - path = (path or "").replace("\\", "/") - if root: - absolute = path if os.path.isabs(path) else os.path.join(os.getcwd(), path) - try: - path = os.path.relpath(absolute, root).replace("\\", "/") - except ValueError: - pass - while path.startswith("./"): - path = path[2:] - return path - - def _ancestors(path: str) -> list[str]: """`.` and every directory above the file, root first.""" parts = path.split("/")[:-1] @@ -207,7 +194,7 @@ def summarize(measures: list[dict[str, Any]], root: str = "") -> dict[str, Any]: if instance.get("file"): files.add(instance["file"]) _tally(by_lane, str(row.get("lane") or "*"), counted) - first = _root_relative(str(instances[0].get("file") or ""), root) + first = root_relative(str(instances[0].get("file") or ""), root) for directory in _ancestors(first): _tally(by_directory, directory, counted) summary: dict[str, Any] = { diff --git a/plugins/code-metrics/scripts/test_pathglob.py b/plugins/code-metrics/scripts/test_pathglob.py index 5021c4f047..b36ce24446 100755 --- a/plugins/code-metrics/scripts/test_pathglob.py +++ b/plugins/code-metrics/scripts/test_pathglob.py @@ -29,6 +29,32 @@ def run(*args: str) -> subprocess.CompletedProcess: ) +class RootRelativeTests(unittest.TestCase): + def test_a_cwd_relative_path_is_rebased_onto_the_root(self) -> None: + import os + import tempfile + + with tempfile.TemporaryDirectory() as tmp: + root = os.path.realpath(tmp) + lib = os.path.join(root, "lib") + os.mkdir(lib) + before = os.getcwd() + os.chdir(lib) + try: + self.assertEqual( + pathglob.root_relative("../plugins/a/x.sh", root), "plugins/a/x.sh" + ) + self.assertEqual(pathglob.root_relative("x.sh", root), "lib/x.sh") + self.assertEqual( + pathglob.root_relative(os.path.join(root, "y.sh"), root), "y.sh" + ) + finally: + os.chdir(before) + + def test_without_a_root_the_path_is_only_normalized(self) -> None: + self.assertEqual(pathglob.root_relative("./a\\b.sh", ""), "a/b.sh") + + class TranslateTests(unittest.TestCase): def test_bare_extension_matches_at_any_depth(self) -> None: self.assertTrue(pathglob.matches("*.sh", "a/b/c.sh")) diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh b/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh index 1a1559dd08..ae052f251d 100755 --- a/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh +++ b/plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh @@ -160,7 +160,7 @@ rc=$? # Merge the pairs the detector reports into clone classes, exclude the declared # replication, recompute the totals from what survived, then state the zero the # recomputation drops when every group was excluded. -"${PY[@]}" "$CLUSTER" <"$WORK/report.json" >"$WORK/clustered.json" || exit 2 +"${PY[@]}" "$CLUSTER" --root "$ROOT" <"$WORK/report.json" >"$WORK/clustered.json" || exit 2 "${PY[@]}" "$FILTER" "${FILTER_ARGS[@]}" <"$WORK/clustered.json" >"$WORK/filtered.json" || exit 2 "${PY[@]}" "$REPORT" resummarize --root "$ROOT" <"$WORK/filtered.json" >"$WORK/summed.json" || exit 2 "${PY[@]}" "$FILTER" --zero-floor --root "$ROOT" <"$WORK/summed.json" >"$WORK/final.json" || exit 2 diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/cluster-clones.py b/plugins/code-metrics/skills/audit-duplication/scripts/cluster-clones.py index f860dfa9b8..007b5fb774 100644 --- a/plugins/code-metrics/skills/audit-duplication/scripts/cluster-clones.py +++ b/plugins/code-metrics/skills/audit-duplication/scripts/cluster-clones.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 """Merge the clone pairs a detector reports into clone classes. - cluster-clones.py [< report.json] + cluster-clones.py [--root ] [< report.json] Reads a `code-metrics/v1` document on stdin and prints it back with the two-instance clone-group rows that share an identical instance merged into one @@ -21,6 +21,10 @@ what the detector said was identical. The merged row keeps the first row's `values`, so the fragment's lines count once, carries the union of the instances sorted by `(file, start_line)`, and appends `clustered` to `labels`. +The sort compares each file made relative to `--root` (the instance keeps the +path the detector gave it), so the first instance, and the directory the +report's rollup attributes the class to, is the same whichever directory the +run started from; without `--root` the paths sort as given. A row with three or more instances is already a class and passes through, as does every row without `instances`, and every row keeps its position. Rows join whatever their `collector`, so a pair another detector reported merges @@ -34,8 +38,12 @@ import json import sys +from pathlib import Path from typing import Any +sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "scripts")) +from pathglob import root_relative # noqa: E402 + MIN_PYTHON = (3, 9) LABEL = "clustered" @@ -48,9 +56,12 @@ def instance_key(instance: dict[str, Any]) -> tuple[str, Any, Any]: ) -def _sort_key(instance: dict[str, Any]) -> tuple[str, int]: +def _sort_key(instance: dict[str, Any], root: str) -> tuple[str, int]: start = instance.get("start_line") - return (instance_key(instance)[0], start if isinstance(start, int) else -1) + return ( + root_relative(instance_key(instance)[0], root), + start if isinstance(start, int) else -1, + ) def _is_pair(row: dict[str, Any]) -> bool: @@ -58,7 +69,7 @@ def _is_pair(row: dict[str, Any]) -> bool: return isinstance(instances, list) and len(instances) == 2 -def cluster(measures: list[dict[str, Any]]) -> list[dict[str, Any]]: +def cluster(measures: list[dict[str, Any]], root: str = "") -> list[dict[str, Any]]: """Return `measures` with pair rows that share an identical instance merged.""" parent: dict[int, int] = { index: index for index, row in enumerate(measures) if _is_pair(row) @@ -97,10 +108,10 @@ def union(left: int, right: int) -> None: if index not in parent: output.append(row) continue - root = find(index) - if root != index: + leader = find(index) + if leader != index: continue - group = members[root] + group = members[leader] if len(group) == 1: output.append(row) continue @@ -109,7 +120,9 @@ def union(left: int, right: int) -> None: for instance in measures[member]["instances"]: instances.setdefault(instance_key(instance), instance) merged = dict(row) - merged["instances"] = sorted(instances.values(), key=_sort_key) + merged["instances"] = sorted( + instances.values(), key=lambda instance: _sort_key(instance, root) + ) labels = [str(label) for label in row.get("labels") or []] if LABEL not in labels: labels.append(LABEL) @@ -119,9 +132,12 @@ def union(left: int, right: int) -> None: def main(argv: list[str]) -> int: - if argv: - print("usage: cluster-clones.py < report.json", file=sys.stderr) + root = "" + if argv == ["--root"] or (argv and argv[0] != "--root") or len(argv) > 2: + print("usage: cluster-clones.py [--root ] < report.json", file=sys.stderr) return 2 + if argv: + root = argv[1] try: document = json.load(sys.stdin) except (json.JSONDecodeError, ValueError) as exc: @@ -134,7 +150,7 @@ def main(argv: list[str]) -> int: return 2 measures = document.get("measures") if isinstance(measures, list): - document["measures"] = cluster(measures) + document["measures"] = cluster(measures, root) print(json.dumps(document, indent=2)) return 0 diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py b/plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py index 683264e6bc..d3147e09ef 100755 --- a/plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py +++ b/plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py @@ -59,6 +59,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "scripts")) from pathglob import matches as glob_matches # noqa: E402 +from pathglob import root_relative # noqa: E402 MIN_PYTHON = (3, 9) CLUSTER_MARKER = " -> " @@ -90,16 +91,7 @@ def read_registry(path: str) -> list[Entry]: def relative(path: str, root: str) -> str: """The instance path root-relative, with forward slashes.""" - path = (path or "").replace("\\", "/") - if root: - absolute = path if os.path.isabs(path) else os.path.join(os.getcwd(), path) - try: - path = os.path.relpath(absolute, root).replace("\\", "/") - except ValueError: - pass - while path.startswith("./"): - path = path[2:] - return path + return root_relative(path, root) def sanctions_plain(entry: str, paths: list[str]) -> bool: diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/test_cluster_clones.py b/plugins/code-metrics/skills/audit-duplication/scripts/test_cluster_clones.py index de65921e96..954931886b 100644 --- a/plugins/code-metrics/skills/audit-duplication/scripts/test_cluster_clones.py +++ b/plugins/code-metrics/skills/audit-duplication/scripts/test_cluster_clones.py @@ -13,6 +13,7 @@ import json import subprocess import sys +import tempfile import unittest from pathlib import Path @@ -186,8 +187,48 @@ def test_stdin_that_is_not_json_is_a_usage_error(self) -> None: self.assertEqual(result.returncode, 2) self.assertIn("not a JSON document", result.stderr) - def test_an_argument_is_a_usage_error(self) -> None: + def test_an_unknown_argument_is_a_usage_error(self) -> None: self.assertEqual(run(document(), "--root").returncode, 2) + self.assertEqual(run(document(), "--registry", "x").returncode, 2) + + def test_root_sorts_instances_by_their_root_relative_path(self) -> None: + # From `lib/`, the dispatcher names the root copy `hook-utils.sh` and + # a plugin copy `../plugins/a/hooks/hook-utils.sh`, which sorts first + # as text; relative to the root the `lib/` copy comes first, so the + # rollup attributes the class the same way a root run does. + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + lib = root / "lib" + lib.mkdir() + doc = document( + pair( + ("../plugins/a/hooks/hook-utils.sh", 1, 41), + ("hook-utils.sh", 1, 41), + ), + pair( + ("../plugins/b/hooks/hook-utils.sh", 1, 41), + ("hook-utils.sh", 1, 41), + ), + ) + result = subprocess.run( + [sys.executable, str(SCRIPT), "--root", tmp], + input=json.dumps(doc), + capture_output=True, + text=True, + cwd=lib, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + rows = measures(result) + self.assertEqual(len(rows), 1) + self.assertEqual( + [i["file"] for i in rows[0]["instances"]], + [ + "hook-utils.sh", + "../plugins/a/hooks/hook-utils.sh", + "../plugins/b/hooks/hook-utils.sh", + ], + ) if __name__ == "__main__": From 94b02fd8f8c3bf1fd0843d4f1076c3240159b164 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 17:48:17 +0000 Subject: [PATCH 13/15] chore(planning): prune the code-metrics duplication audit contract slice Close-out of docs/topics/code-metrics-duplication-audit: the Brief and the dogfood numbers are published in the pull request body, the five-phase plan and the design sketch stay in the branch history (beb6169d:docs/topics/code-metrics-duplication-audit/), the registry grammar is ADR 0033, and the user-facing outcomes are the code-metrics CHANGELOG [0.2.2] entry. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01QShQwS5yDYYToBmofdEY5N --- .../code-metrics-duplication-audit/PLAN.md | 665 ------------------ .../design/design-resolution.md | 93 --- 2 files changed, 758 deletions(-) delete mode 100644 docs/topics/code-metrics-duplication-audit/PLAN.md delete mode 100644 docs/topics/code-metrics-duplication-audit/design/design-resolution.md diff --git a/docs/topics/code-metrics-duplication-audit/PLAN.md b/docs/topics/code-metrics-duplication-audit/PLAN.md deleted file mode 100644 index 333fb7dd0e..0000000000 --- a/docs/topics/code-metrics-duplication-audit/PLAN.md +++ /dev/null @@ -1,665 +0,0 @@ -# Plan: code-metrics audit-duplication fixes - -Closes melodic-software/claude-code-plugins#4068. - -## Brief - -Scope-change note (2026-09-11, before approval, from the fresh-context plan review and the -stress-test, each finding verified against the tree): four criteria below were corrected. The -`hook-utils.sh` class has eighteen instances (root plus seventeen plugins), not seventeen. The -shipped 1mb cap skips one tracked file here, `plugins/miro/dist/index.min.js` (1.45mb), so the -typescript lane reads `partial` by design. The `hook-telemetry-sink.sh` pair differs at line 51 -and has no sync script, so it is not sanctioned replication and its registry line is dropped -(five lines, not six). Pair-to-class merging joins byte-aligned copies; a copy embedded at a -different offset with different surrounding lines stays its own group. Cluster lines carry an -explicit `->` marker because a registered path may contain a space. - -### TLDR - -- `audit-duplication` reports skipped files instead of hiding them: the jscpd adapter pre-filters - by byte size and line count, passes both caps explicitly on 4.x and 5.x, and marks the lane - `partial` when anything was skipped; defaults are no line cap and a 1mb byte cap. -- jscpd's pair reports are merged into clone classes before summarizing, so byte-aligned copies - count once. -- The sanctioned-replication registry gains cluster lines - (` -> ...`) so a canonical file outside any plugin can declare - its copies; this repo's drift checker skips marked lines and five such lines are added here. -- The markdown report sorts clone rows by duplicated lines, adds per-lane and per-directory rollups - (cumulative, counts beside share, computed after exclusion, listed to depth 2), and the JSON gains - `summary.by_lane` / `summary.by_directory` as additive `v1` fields with an explicit ignore-unknown - rule in the schema reference. -- The duplication summary line drops "Functions" and "Over reference"; a run with no detector prints - one consolidated install headline and the skill offers, never performs, the install. Version 0.1.9. - -### Goal - -A whole-tree or change-scoped duplication audit on any repository, this one included, produces -numbers a reader can act on without re-aggregating: no file is silently dropped by a size cap, -byte-aligned copies of a fragment form one group counted once, replication the repository declares -about itself (including copies of a root-level canonical file) is excluded and shown as an -exclusion, and the report says where the surviving duplication sits by lane and by directory. On -this repository the audit reads clean apart from genuine duplication. - -### Constraints - -- The plugin never installs, downloads, or `npx`-fetches a detector; SKILL.md may instruct Claude to - offer the install command and run it only on the user's confirmation. -- The report emits no finding, severity, or exit-code gate; duplication has no reference value. -- The `code-metrics/v1` schema string is unchanged; every JSON change is additive (new optional - fields), and `reference/report-schema.md` states that readers ignore unknown keys. -- Single-token registry lines keep their exact meaning, including a path that contains a space; - the drift checker (`scripts/check-cross-plugin-source-drift.sh`) keeps its behavior for them and - skips only lines carrying the `->` marker. -- Both jscpd 4.x (`latest-4` = 4.3.0) and 5.x (5.2.0) stay supported by the adapter; the adapter - always passes explicit `--max-lines` and `--max-size` on both majors, treats a configured `0` as - `null` (jscpd reads `0` as "default" for 4.x lines and "skip all" everywhere else), and never - emits a `0` cap. -- An unmeasured value is `null`, never `0`; a run that measured nothing keeps "Measured nothing". -- Validate with `scripts/affected-tests.sh --run`; every changed file maps to at least one suite. -- No `lib/hook-utils.sh` or other cross-plugin synced source is edited; every cluster line mirrors - the `src=` and copy list its `scripts/sync-*.sh` already declares. -- One issue, one draft PR whose body opens with `Closes #` and carries the four required - sections; CHANGELOG entry under `[0.1.9]`. - -### Acceptance criteria - -- Running `audit-duplication.sh --json --all` on this repository with jscpd 4.3.0 and again with - 5.2.0 yields, for every byte-identical whole-file class, the same set of instance files per - group; `tokens`, instance order, and the exact line range are excluded from the comparison - because the two majors tokenize differently and hub on different copies (5.x drops a leading - comment block from the clone, so a file that opens with one is reported from its first code line - under 5.x and from line 1 under 4.x; the verifier found one such class, - `resolve-hook.mjs` in two `knowledge` skills). The bash row reads `ok` under both; with the - repository's default `**/dist/**` scope exclusion lifted, the typescript row reads `partial` - under both, naming `plugins/miro/dist/index.min.js` as the one file over the 1mb cap; with the - exclusion in force (this repository's shipped team file) the bundle never reaches the cap and the - row reads `ok`. -- `duplication.max_lines` defaults to `null` (no cap) and `duplication.max_size` to `1mb`; both are - documented in `reference/config.md` (gated against `config-defaults.json`) and exported to the - adapter, and the adapter's tests cover the explicit-cap argv on both majors, the `0`-means-null - rule, and that no `0` cap is ever passed. -- IF every file in a lane is skipped by the caps, THEN that lane's run row reads `partial` with the - reason, no collector is invoked for it, and the script never exits 3 for that cause. -- The eighteen byte-identical copies of `hook-utils.sh` (root `lib/` plus the seventeen plugin - copies `scripts/sync-hook-utils.sh --print-manifest` lists) produce exactly one clone group with - eighteen instances, and with the registry line - `lib/hook-utils.sh -> plugins/*/hooks/hook-utils.sh` that group appears once under `excluded[]` - with `duplicated_lines` counted once. -- Two groups whose instances overlap without identical line ranges (the `hook-telemetry-sink.sh` - shape, and a third copy that carries only part of a fragment two full copies share, so the - detector names the first copy with two different ranges) stay separate groups after the merge. - Three full copies at different line offsets are one class: jscpd pairs each later copy with the - first and names that first copy with the same range in every pair (verified against 5.2.0). -- A cluster line excludes a group only when every instance's root-relative path matches the - canonical path or one of the members (literal or glob) and the instances' directories are all - distinct; two copies inside one directory still count as duplication; a single-token line behaves - exactly as before, a registered path containing a space included; when a single-token line and a - cluster line both match, the first matching line in file order wins. -- `scripts/check-cross-plugin-source-drift.sh --check` passes on this repository with the five new - cluster lines present, each under its own annotation block, and its tests cover a marked line - being skipped. -- After this change, `audit-duplication.sh --json --registry scripts/cross-plugin-source-registry.txt --all` - on this repository reports zero surviving groups whose instances include a root `lib/` file that - a sync script declares as its canonical copy (`scripts/sync-*.sh --print-manifest` `src`), from - the repository root and from a subdirectory alike. Groups over the per-suite test-harness - boilerplate that `lib/*.test.sh` files share with plugin test files survive: no sync script - declares them and the shell-test-helpers convention keeps those helpers per file, so they are - the audit's finding, not sanctioned replication. -- WHILE no registry is configured and none is passed, the report's `excluded[]` is empty and the - summary states that no registry was configured. -- The markdown Measures table for a duplication document lists clone groups in descending order of - duplicated lines, and the report carries a `## Rollup` section with a per-lane table and a - per-directory table (rows to depth 2 by default, `duplication.rollup_depth` configurable) whose - numbers are cumulative up the tree and carry `groups` and `duplicated_lines`; the per-lane - values sum to `summary.duplicated_lines`, and the root row of `by_directory` equals it. -- The JSON `summary` carries `by_lane` and `by_directory` with the same numbers (empty maps when a - duplication collector ran and found nothing); `schema` is still `code-metrics/v1`; - `reference/report-schema.md` documents both fields, the run row's additive `hint` field, and - states that readers ignore unknown keys; `verification:measure`, the one marketplace consumer, - reads only `status` and is unaffected. -- The duplication summary line reads `Files with clones: N.` followed by the duplicated-lines and - exclusion lines, with no "Functions" or "Over reference" text; the sibling skills' summary lines are - byte-identical to today's. -- When no duplication collector resolves, the markdown opens with one headline naming the install - command and `/code-metrics:setup`, taken from the run row's `hint` field rather than parsed out of - its reason, and SKILL.md instructs Claude to offer the install and run it only on confirmation. -- The jscpd adapter's docstring and `reference/collectors.md` state that 4.x and 5.x are both - translated, pin 5.2.0, note the `kind` field, and record that the two majors tokenize differently; - the schema reference names "intentional clones" beside "sanctioned replication". -- `plugin.json` carries the next patch version above the one on the default branch at merge time - (0.2.2 over 0.2.1; the Brief was drafted against 0.1.8) and `CHANGELOG.md` carries the matching - entry covering every item above; `scripts/affected-tests.sh --run` exits 0, or exits 3 with only - Python and Node suites listed as not run, each of which then passes in its own lane, or exits 1 - only for a suite that fails identically on a detached `origin/main` worktree and touches no file - this change edits (named in the PR body). - -### Captured assumptions - -- Version bumps to 0.1.9, not 0.2.0, because this plugin's changelog bumps patch for features - (0.1.7 was a `feat`). Revisit if the marketplace's release convention says a new registry grammar - or JSON fields require a minor bump. -- Once-per-class counting follows PMD CPD and SonarQube; the fetched literature is silent on how - duplicated lines are totalled, and jscpd v5 is the one tool that sums per pair. Revisit if a - standard sets a duplicated-lines definition. -- The merge keys on identical (file, start_line, end_line) instances and equal `lines`, so only - byte-aligned copies join; jscpd 5.x hubs every later copy on the first and names it with the same - range, so full copies at different offsets are one class, while 4.x hubs on the last input and - extends into shared flanking lines, so under 4.x the same copies can stay two groups (a live - probe of three offset copies gave one class under 5.2.0 and two groups under 4.3.0). Closure is exact only for - type-1/type-2 clones, which is all the adapter receives because it passes no `--max-gap-lines`. - Revisit if `similar` clones are ever enabled. -- Merged instances are sorted by root-relative path (`cluster-clones.py --root`) so the first - instance, and therefore `by_directory` attribution, is the same under 4.x (which hubs on the last - input) and 5.x (which hubs on the first), and the same whichever directory the run started from. -- `rollup_depth` default 2 is the plugin's choice; no upstream sets a depth (SonarQube and Codacy - roll up every directory). Revisit if a consuming repository's layout makes depth 2 meaningless. -- The byte cap of 1mb aligns with jscpd 5.0.7's parser guard and SonarJS's 1000kb generated-code - rule, and means 1,048,576 bytes, the value jscpd 5.2.0 reports for `1mb`; no line cap by default - aligns with jscpd 5, PMD CPD, SonarQube, and Linguist. Revisit if jscpd changes its default or - multiplier in a later major. -- The cluster lines live in the existing registry file because the plugin's own documentation names - that file as the shape; this is a repository-convention choice with no external authority. - Revisit if the drift checker grows a second consumer of the file. -- The `->` marker is the cluster-line signal because a registered single-token path may contain a - space and the drift checker's tests protect that case. Revisit if a consuming repository has a - path containing ` -> `. -- The `.claude/hooks/hook-telemetry-sink.sh` and `plugins/claude-ops/hooks/hook-telemetry-sink.sh` - pair differs at line 51 and has no sync script, so the audit keeps reporting its overlap as - duplication; a future sync script earns it a cluster line. Revisit when that script exists. -- "Sanctioned replication" stays the plugin's term; the literature's term "intentional clones" - (Cordy 2008) is named beside it. No upstream tool models the canonical-plus-copies relation, so - no vocabulary conflict exists. -- The upstream doc drift found in jscpd (`docs/rust.md` "no limit" for `--max-size`, "per block" - help text for `--max-lines`, stale jscpd.dev v5 defaults) is reported separately, not here. - -### Out-of-scope - -- Installing a detector on the user's behalf, or an `npx` fallback. -- A pass/fail gate, threshold, or severity for duplication. -- Cross-language clone detection. -- Changing `dupl` or `cpd` adapters beyond passing their rows through the new merge and rollup - steps unchanged. -- A second registry file or a manifest generated from the sync scripts. -- Migrating the registry's single-token lines to the new grammar. -- A repository config excluding `plugins/miro/dist/` from this repo's own audits; the `partial` - row is the designed reading and a config is the consuming repo's choice. - -### Deferred questions - -- None. Every question registered in the interview was answered; no row was deferred or blocked. - -## Plan - -### Goal - -**What**: the six audit-duplication fixes the Brief locks, in `plugins/code-metrics` at 0.1.9, -plus one drift-checker change and five registry cluster lines in this repository, in one draft PR -that closes issue 4068. - -**Why**: the whole-tree audit hid skipped files behind `complete`, inflated every count by pair -reporting, could not declare this repo's own canonical copies, and left the reader to aggregate -by hand; each number the skill reports has to be one a reader can act on. - -### Standards grounding - -No `.claude/standards.yaml` and no `docs/standards/README.md` exist, so the ladder's rung 4 -applied: inferred from repository conventions not auto-loaded. The offer to bootstrap an index -through the planning setup stands; nothing was written. - -| Surface | Sections cited | Layer provenance | -|---|---|---| -| Python | `.claude/rules/ruff-pin.md` (lint only through `scripts/run-ruff.sh check`) | team, path-scoped | -| Skill body | `.claude/rules/skill-bodies-state-current-rules.md` (current rule and reason, no incident narration, `## Next` before `## Gotchas`); `.claude/rules/vendor-docs-are-not-style.md` (house style, `/ai-slop:audit`) | team, path-scoped | -| Shell tests | `docs/conventions/shell-test-helpers/README.md` (per-plugin `pass`/`fail` helpers stay per plugin; repo-tooling suites use `scripts/lib/test-harness.sh`) | team | -| Upstream facts | `docs/conventions/upstream-drift/README.md` (a restated upstream specific carries claim, basis, as-of date, recheck trigger) | team | -| Validation | `AGENTS.md` "Validate a change" (`scripts/affected-tests.sh --run`; exit 3 lists suites in ecosystems it cannot run) | team, ambient | -| Release | `scripts/check-changelog-parity.sh --check-bump` (a manifest bump must add a `## []` entry) | team | - -### Approach - -Five phases, integration slice first. Phase 1 lands the cap machinery end to end (config key to -run row) because it is the only phase that changes what jscpd is asked to scan. Phases 2 and 3 -together satisfy the headline criterion (one eighteen-instance `hook-utils.sh` group, excluded -once). Phase 4 is the rendering layer over the shape Phases 2 and 3 produce. Phase 5 is docs and -release. Phase 3 is committed on its own because editing the registry fans the CI test selection -out to most of the corpus, and a red there should bisect to one commit. - -Build technique: kept tracer-bullet slice (Phase 1's runtime probe runs the audit on this -repository under both jscpd majors); no throwaway spike, since the research and the stress-test -already reproduced every tool behavior the design relies on. - -Pre-flight results (done during planning, recorded so no phase repeats them): the registry file is -named in twelve files; only `scripts/check-cross-plugin-source-drift.sh` parses it into a lookup, -`scripts/check-shell-portability.sh` compares whole lines to a path-within-plugin and cannot match -a marked line, every other mention is a comment. No script outside `plugins/code-metrics/` reads -`code-metrics/v1` documents; `verification:measure` reads `status` only. - -Tool provisioning for probes: jscpd is never installed by the plugin, but the runtime probes and -the fixture captures need both majors. A session runs -`npm install --prefix /jscpd4 jscpd@4.3.0` and `npm install --prefix /jscpd5 jscpd@5.2.0` -into two scratch prefixes outside the repository and prepends the wanted `node_modules/.bin` to -`PATH` per probe. A probe whose major is absent prints `SKIP` and is not a failure, the way -`audit-duplication.test.sh`'s real-cluster case already does. - -### Phase 1: Explicit caps, adapter pre-filter, partial run row [DONE] - -Review: code-design - -1. Add `duplication.max_lines` (`null`), `duplication.max_size` (`"1mb"`), and - `duplication.rollup_depth` (`2`) to `scripts/config-defaults.json`; add the three rows to - `reference/config.md` (gated by `scripts/check-code-metrics-config-reference.py`), with the - note that a `0` cap means `null` and that CRLF checkouts count one extra byte per line; add the - three keys to `skills/setup/templates/config-template.yaml` (pinned by `test_setup_apply.py`). -2. `audit-duplication.sh`: read the caps beside the existing tunables, map `0` to null, and export - `CODE_METRICS_DUP_MAX_LINES` (empty when null) and `CODE_METRICS_DUP_MAX_SIZE`; keep the - existing three exports byte-identical. -3. `dispatch.sh`: before each `collect`, export `CODE_METRICS_PARTIAL_REASON_FILE` pointing at - `$WORK/partial.$lane.$measure.$tool` (the work dir is a fresh `mktemp -d` per run and one tool - runs per lane, so no stale file exists); after a `collect` that exits 0, if that file is - non-empty, write the run row as `partial` with the file's single line as the reason, else `ok` - as today. On a failed probe, also write the adapter's install hint into an additive `hint` - field on the run row (`null` otherwise) and keep `reason` as it is built today. - [EXEC-SHAPE] File-per-channel matches how `dispatch.sh` already isolates each adapter's stdout - and stderr into `$WORK` files. -4. `collectors/jscpd.py`: parse `CODE_METRICS_DUP_MAX_SIZE` with the multipliers jscpd uses - (`kb` = 1024, `mb` = 1,048,576, bare digits = bytes) and `CODE_METRICS_DUP_MAX_LINES`; treat an - empty or `0` value as no cap; pre-filter the file list by `os.stat` size and, only when a line - cap is set, by a binary-mode newline count; pass `--max-size ` always and - `--max-lines ` always (the review raised the sentinel from - 1000000, which a generated file can reach, to the signed 32-bit maximum both majors accept), so - the pre-filter is the only gate on both - majors [EXEC-SHAPE] (4.x reads `--max-lines 0` as "use the 1000 default" and both majors read a - `0` size as "skip all"); when files were skipped, write one line to the partial-reason file - (`N of M files skipped by duplication.max_size / max_lines ; largest: ()`), - or to stderr when the variable is unset (direct runs) or the path is unwritable, never failing - for that; when the pre-filter leaves zero files, write the note and return 0 without invoking - jscpd (4.x would write no report and the current exit-3 path would call that a failure). Extend - the module docstring: 4.x and 5.x both translate; drop the "jscpd 4 is not translated" - sentence; record the 5.2.0 `kind` field and that the majors tokenize differently. -5. `registry-filter.py --zero-floor`: a `partial` duplication row counts as measured, so an - all-excluded or clone-free lane that skipped a file still states `duplicated_lines: 0` and - `clone_groups: 0`. -6. `reference/collectors.md`: jscpd row pinned to 5.2.0 with a fresh as-of date; add four-part - verification records for the 4.x maintenance line (`latest-4` = 4.3.0), the `1mb` multiplier, - and the token-count difference between majors. - -**Files Affected** - -| File | Action | What changes | -|---|---|---| -| `plugins/code-metrics/scripts/config-defaults.json` | Modify | three keys under `duplication` | -| `plugins/code-metrics/reference/config.md` | Modify | three key rows, `0`-means-null and CRLF notes | -| `plugins/code-metrics/skills/setup/templates/config-template.yaml` | Modify | three keys | -| `plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh` | Modify | two exports, `0` to null | -| `plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh` | Modify | export assertions; its `5.1.2` stub version string becomes `5.2.0` | -| `plugins/code-metrics/scripts/dispatch.sh` | Modify | partial-reason channel, `partial` row, `hint` field | -| `plugins/code-metrics/scripts/dispatch.test.sh` | Modify | partial-row case: sets `CODE_METRICS_DUP_MAX_SIZE` to a small value itself (dispatch never exports caps) and adds a jscpd stub beside the existing scc stub; a `hint` assertion on a failed probe | -| `plugins/code-metrics/scripts/collectors/jscpd.py` | Modify | caps, pre-filter, note, docstring | -| `plugins/code-metrics/scripts/collectors/test_jscpd.py` | Modify | argv on both stub versions, skip, all-skipped, size grammar, `0`, unset variable; docstring version | -| `plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py` | Modify | zero floor counts `partial` | -| `plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py` | Modify | zero-floor `partial` case | -| `plugins/code-metrics/reference/collectors.md` | Modify | jscpd row, three verification records | - -**Sanity Check:** - -- `python3 scripts/check-code-metrics-config-reference.py` exits 0; - `python3 -m unittest plugins/code-metrics/skills/setup/scripts/test_setup_apply.py` exits 0. -- `python3 -m unittest plugins/code-metrics/scripts/collectors/test_jscpd.py` exits 0; its argv-log - case asserts `--max-size 1048577` and `--max-lines 2147483647` present and no argument equal to `0` - follows either flag. -- `bash plugins/code-metrics/scripts/dispatch.test.sh` exits 0 with a case whose JSON has a run row - `status == "partial"` and a reason matching `^[0-9]+ of [0-9]+ files skipped`, and a case whose - failed-probe row carries a non-null `hint`. -- `python3 -m unittest plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py` - exits 0 with the zero-floor `partial` case. -- Runtime probe (SKIP when a major is absent): with jscpd 5.2.0 on PATH, - `audit-duplication.sh --json --all | jq -c '[.run[]|select(.measure=="duplication")|{lane,status}]'` - shows `bash` `ok` and `typescript` `partial`, and the typescript reason names - `plugins/miro/dist/index.min.js`; the same under 4.3.0. - -### Phase 2: Merge pairs into clone classes [DONE] - -Review: code-design - -1. New `skills/audit-duplication/scripts/cluster-clones.py` (stdin document, stdout document): - union-find over every row with exactly two `instances[]`, whatever its `collector` (a - three-or-more-instance row is already a class and passes through); two rows join when they - share an instance with identical `(file, start_line, end_line)` and equal `values.lines`; the - merged row keeps the first row's `values`, the union of instances sorted by `(file, - start_line)`, and appends `clustered` to `labels`. Rows without `instances` pass through. Exit 0 - on a printed document, 2 on a non-JSON stdin. -2. `audit-duplication.sh`: pipe `report.json` through `cluster-clones.py` before `registry-filter.py`. -3. Fixtures, outside `fixtures/sources` so no suite that scopes that tree changes its counts: - `scripts/fixtures/clone-classes/aligned/{a,b,c}/shared/shared-utils.sh` (three byte-identical - copies) and `scripts/fixtures/clone-classes/offset/{c1,c2,c3}.sh` (`c1` and `c2` carry one 34-line - fragment at different line offsets with different flanking lines; `c3` carries only the first - 24 lines of it, so its pair names `c1` with a shorter range and the two pairs do not merge); - committed captures - `scripts/fixtures/tool-output/jscpd-aligned3.json` and `jscpd-offset3.json` produced by a real - jscpd 5.2.0 run, then rewritten to repo-relative names (the adapter passes `--absolute`, so the - raw capture carries machine paths, and the stub replays the file regardless of input). -4. `test_cluster_clones.py`: aligned three copies collapse to one three-instance group with - `lines` counted once; offset copies stay two groups; a three-instance input row passes - through; a two-instance `cpd`-labelled row joins when it shares an identical instance; merged - instance order is by path; `summary` is left to `report.py resummarize`. - -**Files Affected** - -| File | Action | What changes | -|---|---|---| -| `plugins/code-metrics/skills/audit-duplication/scripts/cluster-clones.py` | Create | the post-pass | -| `plugins/code-metrics/skills/audit-duplication/scripts/test_cluster_clones.py` | Create | output-based tests | -| `plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh` | Modify | pipeline step | -| `plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh` | Modify | aligned and offset cases through the stub | -| `plugins/code-metrics/scripts/fixtures/clone-classes/aligned/{a,b,c}/shared/shared-utils.sh` | Create | three copies | -| `plugins/code-metrics/scripts/fixtures/clone-classes/offset/{c1,c2,c3}.sh` | Create | offset copies | -| `plugins/code-metrics/scripts/fixtures/tool-output/jscpd-aligned3.json` | Create | capture, relative names | -| `plugins/code-metrics/scripts/fixtures/tool-output/jscpd-offset3.json` | Create | capture, relative names | -| `plugins/code-metrics/scripts/dispatch.test.sh` | KEEP | its `summary.files == 7` assertion over `fixtures/sources` is untouched by the new fixture tree | - -**Sanity Check:** - -- `python3 -m unittest plugins/code-metrics/skills/audit-duplication/scripts/test_cluster_clones.py` exits 0. -- `bash plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh` exits 0 - with a case asserting `summary.clone_groups == 1` and `len(measures[0].instances) == 3` on the - aligned capture with no registry, and `summary.clone_groups == 2` on the offset capture. -- `cmp` the three aligned copies pairwise: identical; - `grep -c '^/' plugins/code-metrics/scripts/fixtures/tool-output/jscpd-aligned3.json` prints `0`. -- `bash plugins/code-metrics/scripts/dispatch.test.sh` exits 0 unchanged. - -### Phase 3: Registry cluster lines, drift checker, this repo's five lines [DONE] - -Review: code-design - -Committed on its own (the registry edit fans CI's test selection out to roughly 225 suites). - -1. `registry-filter.py`: a line containing ` -> ` is a cluster line: the text before the arrow is - the canonical path, the whitespace-separated tokens after it are members (literal paths or - `pathglob` globs); every other non-comment line is a single token taken whole, spaces included. - Before matching, normalize every instance path to root-relative by joining the cwd-relative - value onto the cwd and taking `relpath` against `--root` (the dispatcher rebases paths onto the - cwd, and an anchored glob rejects a `../` prefix). A cluster line sanctions a group when every - normalized instance equals the canonical path or matches one member, and the instances' - `dirname`s are pairwise distinct (the glob matcher anchors the whole path, so "prefix before the - token" is empty for a glob and the single-token prefix rule cannot be reused). `excluded[].path` - carries the line text. The first matching line in file order wins, stated in the docstring and - `reference/config.md`. -2. `scripts/check-cross-plugin-source-drift.sh`: in the registry load loop, `continue` on a line - containing ` -> `, with a comment naming the cluster-line grammar and its reader. - `check-cross-plugin-source-drift.test.sh`: a marked line neither registers nor reports - `REGISTRY STALE`; the existing space-bearing-path case stays green. -3. `scripts/cross-plugin-source-registry.txt`: header line "One path-within-plugin per line" - gains the cluster-line sentence; five cluster lines, each under its own annotation block naming - its dedicated check (the production-registry policy test resets its comment block after every - entry): `lib/hook-utils.sh -> plugins/*/hooks/hook-utils.sh` (`scripts/sync-hook-utils.sh --check`); - `lib/rewrite-guard.sh -> plugins/*/hooks/rewrite-guard.sh` (`scripts/sync-rewrite-guard.sh --check`); - `lib/index-regen.sh -> plugins/*/scripts/index-regen.sh` (`scripts/sync-index-regen.sh --check`); - `lib/resolve-convention-pattern.sh -> plugins/*/hooks/resolve-convention-pattern.sh` - (`scripts/sync-resolve-convention-pattern.sh --check`); - `lib/parse-concern-value.sh -> plugins/*/skills/*/scripts/parse-concern-value.sh plugins/*/skills/*/scripts/lib/parse-concern-value.sh` - (`scripts/sync-parse-concern-value.sh --check`). Before committing, run the Phase 3 probe below - and add a line only for a surviving root `lib/` group that a sync script declares. -4. Fixture registry `scripts/fixtures/registry/cluster.txt` gains a commented cluster-line example - and its header sentence; `test_registry_filter.py` gains: a cluster line excludes a - canonical-plus-copies group; a glob member matches; two instances in one directory keep the - group; a single-token path containing a space still matches whole; first matching line wins - when both shapes match; instances given cwd-relative from a subdirectory still match. -5. Prose that restates the grammar: `reference/config.md` `duplication.registries` row, - `plugins/claude-config/skills/audit-pass/reference/exclusion-set.md` line 18 ("entries are paths - within each plugin"), and the SKILL.md configuration paragraph (Phase 5). - -**Files Affected** - -| File | Action | What changes | -|---|---|---| -| `plugins/code-metrics/skills/audit-duplication/scripts/registry-filter.py` | Modify | cluster grammar, root normalization, precedence | -| `plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py` | Modify | six cases | -| `plugins/code-metrics/scripts/fixtures/registry/cluster.txt` | Modify | example line, header | -| `scripts/check-cross-plugin-source-drift.sh` | Modify | skip marked lines | -| `scripts/check-cross-plugin-source-drift.test.sh` | Modify | one case | -| `scripts/cross-plugin-source-registry.txt` | Modify | header, five annotated cluster lines | -| `plugins/code-metrics/reference/config.md` | Modify | registries row text | -| `plugins/claude-config/skills/audit-pass/reference/exclusion-set.md` | Modify | one sentence | - -**Sanity Check:** - -- `python3 -m unittest plugins/code-metrics/skills/audit-duplication/scripts/test_registry_filter.py` exits 0. -- `bash scripts/check-cross-plugin-source-drift.sh --check` exits 0 on this tree; - `bash scripts/check-cross-plugin-source-drift.test.sh` exits 0 (including the production-registry - policy case). -- Runtime probe (SKIP when absent) with jscpd 5.2.0 on PATH, from the repository root: - `audit-duplication.sh --json --registry scripts/cross-plugin-source-registry.txt --all | jq '[.measures[]|select(any(.instances[]; .file|test("^lib/[^/]*[.]sh$") and (.file|test("[.]test[.]sh$")|not)))]|length'` - prints `0` (the surviving `lib/` groups are all `lib/*.test.sh` harness boilerplate), and `jq '[.excluded[]|select(.path|startswith("lib/hook-utils.sh"))]|length'` - prints `1` with that entry's `instances` length equal to - `$(scripts/sync-hook-utils.sh --print-manifest | grep -c copy) + 1`; the same two commands run - from `plugins/code-metrics` with `--registry ../../scripts/cross-plugin-source-registry.txt` - print the same values. - -### Phase 4: Report sort, rollups, additive summary fields, summary line, no-detector headline [DONE] - -Review: code-design - -1. `report.py summarize` gains an optional `--root`: when clone-group rows exist, add `by_lane` - (lane to `{groups, duplicated_lines}`) and `by_directory` (every ancestor directory of each - group's first instance after root-normalization, cumulative, root as `.`, same shape); emit - both as empty maps when a duplication collector ran and no group survived. `assemble` and - `resummarize` accept `--root`; `audit-duplication.sh` passes it to `resummarize`, so the maps - are computed over surviving groups after exclusion. A group is attributed to its first - instance's ancestors (instances are path-sorted by Phase 2), so the identity that holds is - `by_directory["."]["duplicated_lines"] == summary.duplicated_lines` and the per-lane sum; rows - below the root cannot be summed, which the schema reference states. -2. `report.py assemble`: a `partial` run row counts as measured for document `status`, so an - all-skipped lane yields `partial`, not `empty`, and the `Unavailable:` line is followed by a - `Partial:` line naming lanes that skipped files. -3. `report.py render`: for a document with clone-group rows, sort measures by `values.lines` - descending, then `tokens`, then first instance path (other skills keep today's sort); add a - `## Rollup` section after Measures with a per-lane table and a per-directory table listing - directories whose depth is at most `--rollup-depth` (default 2; `audit-duplication.sh` passes - the resolved key); render the summary line as `Files with clones: N.` when the document is - duplication-shaped (any `instances[]` row or `skill == "audit-duplication"`), otherwise today's - line byte for byte; when a duplication document has an empty `excluded[]`, print - `Excluded by a sanctioned-replication registry: 0 (no registry configured).`; when every - `duplication` run row is `unavailable`, emit one headline under the title with the first - non-null `hint` and `/code-metrics:setup`, and print each lane row's reason unchanged. -4. `reference/report-schema.md`: document `by_lane`, `by_directory` (attribution rule, root - identity, empty-map floor), the run row's `hint`, the `partial` document status for a lane that - skipped files; add the sentence that readers ignore unknown keys; note "intentional clones" - beside "sanctioned replication" in the `excluded` row. -5. `test_report.py`: rollup sums and root identity; cumulative ancestors; depth cut in markdown - only; empty maps on a clone-free duplication run; sort order; summary line per skill (the - existing exact-dict assertion for a size document stays untouched); no-registry sentence; - no-detector headline once; `partial` document status; a size document renders byte-identically. - -**Files Affected** - -| File | Action | What changes | -|---|---|---| -| `plugins/code-metrics/scripts/report.py` | Modify | rollups, `--root`, status, sort, summary line, headline | -| `plugins/code-metrics/scripts/test_report.py` | Modify | nine cases | -| `plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.sh` | Modify | pass `--root` and `--rollup-depth` | -| `plugins/code-metrics/reference/report-schema.md` | Modify | fields, status, ignore-unknown rule, term | - -**Sanity Check:** - -- `python3 -m unittest plugins/code-metrics/scripts/test_report.py` exits 0. -- `audit-duplication.sh --all | grep -c 'Functions:'` prints `0` and - `audit-duplication.sh --all | grep -c '^Files with clones:'` prints `1`; - `audit-size.sh --all | grep -c 'Functions:'` prints `1`. -- `audit-duplication.sh --json --all | jq '([.summary.by_lane[].duplicated_lines]|add) == .summary.duplicated_lines and .summary.by_directory["."].duplicated_lines == .summary.duplicated_lines'` - prints `true`. -- `PATH= audit-duplication.sh --all | grep -c 'No clone detector ran'` prints `1`; - the install hint appears in that headline and again in each lane row's reason, which the - dispatcher builds with the hint embedded and this phase leaves unchanged. - -### Phase 5: SKILL.md, README, CHANGELOG, version, dogfood [DONE] - -1. `skills/audit-duplication/SKILL.md`: configuration section names the three new keys, the - `0`-means-null rule, and both registry line shapes; "Run it" gains the no-detector instruction - (offer the install command to the user, run it only on confirmation, never silently); "Reading - the numbers" states clone classes (byte-aligned copies merge, offset copies stay separate), the - `partial` row and document status, the rollup and its root identity; the pairs gotcha is - rewritten; `## Next` stays before `## Gotchas`. Prose in house style (`/ai-slop:audit` on the - file). -2. `README.md`: the audit-duplication row mentions clone classes, rollups, and the cluster line; - the known-gaps bullet's list of prose-restated defaults is checked against what SKILL.md now - restates. -3. `CHANGELOG.md`: `## [0.1.9]` with Added / Changed / Fixed entries, one per Brief item, including - the offset-copies limitation and the `partial` reading; `.claude-plugin/plugin.json` version 0.1.9. -4. Dogfood: `scripts/affected-tests.sh --run` over the whole diff (exit 0, or exit 3 whose `NOT - RUN` list is Python suites, each then run with `python3 -m pytest`); `scripts/run-ruff.sh check` - over the changed Python; `shellcheck` and `shfmt -d` over changed shell; both jscpd majors' - whole-tree runs recorded here as distilled numbers (surviving groups, duplicated lines, - exclusions, partial lanes) with no memory-slice paths. - -**Files Affected** - -| File | Action | What changes | -|---|---|---| -| `plugins/code-metrics/skills/audit-duplication/SKILL.md` | Modify | config, run-it, reading, gotchas | -| `plugins/code-metrics/README.md` | Modify | row, known gaps | -| `plugins/code-metrics/CHANGELOG.md` | Modify | `[0.1.9]` | -| `plugins/code-metrics/.claude-plugin/plugin.json` | Modify | version | - -**Sanity Check:** - -- `jq -r .version plugins/code-metrics/.claude-plugin/plugin.json` prints `0.1.9`; - `bash scripts/check-changelog-parity.sh --check-bump origin/main` exits 0. -- `scripts/affected-tests.sh --run` exits 0, or exits 3 with every `NOT RUN` line naming a - `test_*.py` that then passes under `python3 -m pytest`; `scripts/run-ruff.sh check plugins/code-metrics scripts/check-code-metrics-config-reference.py` exits 0. -- `grep -n '^## Next' plugins/code-metrics/skills/audit-duplication/SKILL.md` precedes `^## Gotchas`; - `grep -c '5\.1\.2' plugins/code-metrics/reference/collectors.md plugins/code-metrics/scripts/collectors/test_jscpd.py plugins/code-metrics/skills/audit-duplication/scripts/audit-duplication.test.sh` prints `0` for each. -- `markdownlint-cli2` over the changed markdown exits 0. - -**Dogfood (whole tree, `--all --registry scripts/cross-plugin-source-registry.txt`, 2026-09-11):** - -| Detector | Status | Surviving classes | Duplicated lines | Files with clones | Excluded | Partial lanes | -|---|---|---|---|---|---|---| -| jscpd 5.2.0 | partial | 742 | 11831 | 487 | 14 (the five cluster lines: 18, 7, 4, 2, 2 instances) | typescript (1 of 295 files over 1mb: `plugins/miro/dist/index.min.js`) | -| jscpd 4.3.0 | partial | 625 | 10926 | 429 | 14 | typescript (same file) | - -Before this change the same 5.2.0 run reported 919 pair rows and 75267 duplicated lines with no -skipped file named. After the merge with the 0.2.1 line of the plugin (default `scope.exclude` -globs, the `other` lane, `scope.registries` in this repository's team file) the 5.2.0 run reads -`complete` with 723 classes, 11378 duplicated lines, 492 files with clones, and 16 exclusions: the -miro bundle now falls under the default `**/dist/**` exclusion before the cap is reached, so no -lane is `partial`; pass `--all` on a tree without that exclusion, or set `scope.exclude: []`, to -see the cap and the `partial` row. The two majors tokenize differently, so their counts are not comparable with -each other. The largest surviving classes are per-suite test-harness boilerplate (`pass`/`fail` -helpers shared by `lib/*.test.sh` and plugin test files), which no sync script declares and the -shell-test-helpers convention keeps per file: a finding for the operator, not sanctioned -replication. - -### Alternatives Considered - -| Alternative | Why rejected | Switch condition | -|---|---|---| -| Detect skips as files-passed minus `statistics.total.sources` | contradicted: `sources` counts token sources that reached detection, reproduced on 4.3.0 and 5.2.0 | jscpd adds a per-file skip list to its JSON report | -| Keep jscpd's per-major defaults, detect only | jscpd 4.x users silently lose every file over 1000 lines | jscpd 4.x line reaches end of life | -| Any whitespace marks a cluster line | the drift checker's tests protect a registered path containing a space | the registry documents a no-spaces rule for paths | -| A tab or a leading sigil as the cluster marker | less readable than `->` in a hand-edited file; equally unambiguous | a consuming repository has a path containing ` -> ` | -| Merge pairs inside the jscpd adapter | a second pair-reporting collector would need it again; the rule is about the report | no other collector ever reports pairs | -| A second registry file for cluster lines | two registries for one concept; the plugin's docs already name this file as the shape | a second reader of the registry file cannot be taught to skip marked lines | -| `by_directory` as direct-parent rows | no documented precedent; a plugin's files would never roll up to the plugin | a consumer needs non-overlapping per-directory sums | -| Stderr prefix as the adapter-to-dispatcher channel | `dispatch.sh` reads stderr only on failure and truncates it; a file is unambiguous and output-testable | the adapter contract grows a structured stderr protocol | -| Strip the install hint out of the run row's reason for the headline | the hint itself contains parentheses and multi-tool lanes concatenate several | never; an additive `hint` field is strictly simpler | -| Exclude `plugins/miro/dist/` through a repo config so no file is skipped here | hides the `partial` reading the change exists to produce; a config is the consuming repo's choice | the repo adopts a `.claude/code-metrics.yaml` for other reasons | - -### Test Strategy - -Output-based tests drive every script at its command line with fixture inputs and assert on the -printed document, per the TDD principles skill's decision order (output first, state second, -communication last). jscpd is the one unmanaged out-of-process dependency and stays stubbed by a -fake binary that replays a committed capture; nothing in-process is mocked: `cluster-clones.py`, -`registry-filter.py`, and `report.py` are driven for real by `audit-duplication.test.sh` and -`dispatch.test.sh`. One communication-based assertion exists because the boundary is the tool's -argv: `test_jscpd.py`'s existing `argv_log` stub asserts the explicit caps. Red first: each phase -writes its failing test against the named boundary, then the change. - -Test boundaries (all existing unless marked): `jscpd.py collect` argv and partial-reason output -(existing CLI, new env vars); `dispatch.sh` run rows (existing, new `hint` field); -`cluster-clones.py` stdin/stdout (new script, same document contract as `registry-filter.py`); -`registry-filter.py --registry --root` (existing); `report.py summarize|resummarize|render` -(existing, new `--root` and `--rollup-depth` arguments); `audit-duplication.sh --json` (existing); -`check-cross-plugin-source-drift.sh --check` (existing). A boundary implementation picks that this -list does not name is a deviation logged to `DEVIATIONS.md` beside this file. - -Edge cases named in the Brief's criteria: offset copies; overlapping-but-not-identical ranges; two -copies in one directory; a space-bearing single-token path; a subdirectory cwd; all files skipped; -a `0` cap; no registry configured; a lane whose collector never resolved; a clone-free duplication -run. Existing tests updated: `audit-duplication.test.sh`, `test_jscpd.py`, -`check-cross-plugin-source-drift.test.sh`, `test_registry_filter.py`, `dispatch.test.sh`; -`test_setup_apply.py` passes unchanged once the template carries the keys. - -### Risks and Mitigations - -| Risk | Likelihood | Impact | Mitigation | -|---|---|---|---| -| jscpd 4.x argv differs from 5.x for a passed cap | Low | Med | both majors verified this session (`--max-lines`, `--max-size` share names; raw byte counts accepted); Phase 1's probe runs both | -| A marked registry line breaks a reader not found in pre-flight | Low | High | pre-flight grepped all 12 mentions; the drift-checker test case is the guard; the registry edit selects every suite referencing it | -| The union-find joins two different fragments | Low | Med | the key requires identical range and equal `lines`; the offset fixture asserts two groups | -| `by_directory` differs by jscpd major | Low | Low | merged instances are path-sorted; the probe compares instance sets, not order | -| The sort or summary change alters other skills' markdown | Low | Med | both branches are gated on clone rows; `test_report.py` asserts a size document renders byte-identically | -| Full-corpus CI run on the registry commit hides an unrelated red | Med | Low | Phase 3 is its own commit; `--check` and its test run locally before the push | -| Version bump without changelog entry fails CI | Low | Low | Phase 5 sanity check runs the parity gate | - -## Blast radius - -MEDIUM. About thirty files across one plugin and two repo tooling scripts; one CI gate script and -the registry it reads change, which fans `scripts/affected-tests.sh`'s selection out to most of -the corpus; every change is a revertable commit on a feature branch; the config reference gate, -the changelog parity gate, the drift checker's own test, and the plugin's suites cover it. - -## Stress-test summary - -Both passes ran on the first draft in fresh contexts. The plan reviewer returned 2 CRITICAL, 9 -IMPORTANT, 8 SUGGESTION; `/planning:devils-advocate` returned 1 CRITICAL, 7 HIGH, 6 MEDIUM, 3 LOW, -with probes against both jscpd majors. Every load-bearing finding was verified against the tree -before being applied: the eighteen-instance count; the 1.45mb minified file in the typescript -lane; the non-identical `hook-telemetry-sink.sh` pair with no sync script; the drift test's -annotation-block-per-entry policy and its protected space-bearing path; the zero floor keyed on -`ok`; the cwd rebase of instance paths; the hint glued into the reason string; the whole-path -anchoring of the glob matcher; jscpd's star-shaped pairs with the hub on the last input (4.x) or -the first (5.x); offset copies producing ranges that differ by one line; the fixture-count -assertion in `dispatch.test.sh`; and `affected-tests.sh`'s exit-3 contract. No research-iterate -round was needed: every contested claim was settled by a probe the reviewing agent ran and this -session reproduced. - -## Execution shape - -Fully sequential: 1 → 2 → 3 → 4 → 5. `audit-duplication.sh` is edited in Phases 1, 2, 3 and 4, -`registry-filter.py` in Phases 1 and 3, and Phase 4 renders the shape Phases 2 and 3 produce, so -no two phases are file-disjoint and no parallel wave exists. All-main-session execution. - -| Phase | Surface | Basis | -|---|---|---| -| 1 | main-session | adapter, dispatcher, and config edits interlock; the runtime probe needs the session's scratch jscpd prefixes | -| 2 | main-session | small new script plus captures that must be rewritten by hand to relative names | -| 3 | main-session | a CI gate script and the registry change together and are committed alone | -| 4 | main-session | one shared renderer; judgment on byte-identical sibling output | -| 5 | main-session | prose in house style, release bump, dogfood numbers recorded in this file | - -## Open questions - -None at approval time beyond the gates below. - -## Handoff to implementation - -### User-approval gates - -- The four Brief corrections in the scope-change note at the top of the Brief (instance count, - the `partial` typescript row, the dropped sixth registry line, the byte-aligned merge caveat) - and the `->` marker, which amends the grammar the interview locked; approving this plan approves - them. Any later change to an acceptance criterion stops and asks. - -### Execution shape ([EXEC-SHAPE] tagged) - -- Sequential phases with Phase 3 as its own commit; per-phase sanity checks as written; the - partial-reason file channel; the large explicit `--max-lines` and the bound-plus-one byte cap - passed to jscpd so the adapter's pre-filter is the only gate; fixtures placed outside - `fixtures/sources`; scratch-prefix jscpd installs for probes with `SKIP` when absent. - -### Mechanical work - -- One commit per phase (Phase 3 alone), each carrying its `[DONE]` tag flip in this file; run the - phase's sanity checks before committing; push after each commit. Sequential fallback is not - needed (no parallel wave). At PR time, run `/planning:plan close-out`. diff --git a/docs/topics/code-metrics-duplication-audit/design/design-resolution.md b/docs/topics/code-metrics-duplication-audit/design/design-resolution.md deleted file mode 100644 index 1deb6610ed..0000000000 --- a/docs/topics/code-metrics-duplication-audit/design/design-resolution.md +++ /dev/null @@ -1,93 +0,0 @@ -# Design resolution: code-metrics audit-duplication fixes - -outcome: early-exit -tier: B (light design: localized contract additions inside one plugin, no new module, no topology change) -resolved: 2026-09-11, by the interview (two rounds) and five verified research runs - -## Why early-exit - -Every contract this change adds is an additive extension of a shape the plugin already has, and -the interview locked each one with its rationale and sources. No thread is open that a design -session would resolve differently from the Brief. The type sketch below is what an implementer -needs; `/planning:plan` consumes it. - -## Type sketch - -### Registry line grammar (extends `registry-filter.py`) - -```text -line := comment | blank | single | cluster -single := path-within-plugin # whole line, spaces included; unchanged -cluster := canonical-path " -> " member (SP+ member)* # the arrow is the marker -member := repo-relative path | glob (pathglob.py syntax) -``` - -Instance paths are normalized to root-relative first (the dispatcher emits them cwd-relative). A -group is excluded by a `cluster` line when every normalized instance equals the canonical path or -matches one member, and the instances' `dirname`s are pairwise distinct (the glob matcher anchors -the whole path, so the single-token line's "prefix before the suffix" rule does not transfer). The -first matching line in file order wins. The `excluded[]` record keeps -`{registry, line, path, instances}` with `path` = the line's text. - -### Clone-group row after clustering (unchanged schema, N instances) - -```json -{"file": null, "function": null, "lane": "bash", - "instances": [{"file": "...", "start_line": 1, "end_line": 3136}, "... N entries"], - "values": {"lines": 3136, "tokens": 25137}, - "collector": "jscpd", "labels": ["token-based", "clustered"]} -``` - -Merge key: two pair rows join when they share an instance with identical `(file, start_line, -end_line)` and equal `values.lines`. Union-find over all pair rows; `tokens` taken from the first -pair. Rows from `dupl` and `cpd` pass through untouched (already N-ary). - -### Run row for a cap-skipped lane - -```json -{"lane": "bash", "measure": "duplication", "collector": "jscpd 4.3.0", "status": "partial", - "reason": "3 of 412 files skipped by duplication.max_size 1mb / max_lines none; largest: lib/x.sh (2.1mb)"} -``` - -Adapter to dispatcher channel: the adapter writes the skip note to the path in -`CODE_METRICS_PARTIAL_REASON_FILE` (set by `dispatch.sh` per lane/measure/tool, in a work dir -that is fresh per run); when the file is non-empty after a successful collect, `dispatch.sh` writes -the run row as `partial` with that text. When the variable is unset the note goes to stderr and is -never a failure. When the pre-filter leaves zero files, the adapter writes the note and exits 0 -without invoking the tool; the row is `partial` and no `exit 3` occurs. A failed probe's run row -carries the adapter's install hint in an additive `hint` field beside the unchanged `reason`. - -The merged clone-group row's instances are sorted by `(file, start_line)`, so the first instance -is the same whichever jscpd major produced the pairs (4.x hubs on the last input, 5.x on the first). - -### Summary additions (additive `code-metrics/v1`) - -```json -"summary": {"files": 432, "functions": 0, "over_reference": {}, - "duplicated_lines": 16498, "clone_groups": 705, - "by_lane": {"bash": {"groups": 380, "duplicated_lines": 12450}, "...": {}}, - "by_directory": {"plugins/code-metrics": {"groups": 127, "duplicated_lines": 1867}, "...": {}}} -``` - -`by_directory` keys are every ancestor directory of a group's first instance up to the root -(cumulative), rendered in markdown to `duplication.rollup_depth` (default 2). Both maps are -computed from surviving groups, after registry exclusion. Readers ignore unknown keys (stated in -`reference/report-schema.md`). - -### Configuration keys (`config-defaults.json`, `reference/config.md`, setup template) - -```yaml -duplication: - min_tokens: 50 - min_lines: 5 - ignore: [] - registries: [] - max_lines: null # no line cap; a number is a plugin-local guard - max_size: 1mb # jscpd 5.0.7 parser guard; SonarJS 1000kb generated-code rule - rollup_depth: 2 -``` - -Exported to adapters as `CODE_METRICS_DUP_MAX_LINES` (empty when null) and -`CODE_METRICS_DUP_MAX_SIZE`; the jscpd adapter passes both explicitly on every major, translating a -null line cap to a large explicit value on 4.x (whose `0` means "use the 1000 default") and never -emitting `--max-size 0`. From d91a59717adf216fcd2e107972af09d4edd61a45 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 18:00:53 +0000 Subject: [PATCH 14/15] fix(code-metrics): hygiene for the duplication audit change CI's lint lane on the pull request head reported three hygiene failures: the two new Python scripts carried a shebang without the exec bit, the three regenerated jscpd captures ended without a newline, and the audit-duplication description had grown past the Agent Skills field maximum of 1024 codepoints. The description now states the same triggers and scope in 963. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01QShQwS5yDYYToBmofdEY5N --- .../scripts/fixtures/tool-output/jscpd-aligned3.json | 2 +- .../scripts/fixtures/tool-output/jscpd-offset3.json | 2 +- plugins/code-metrics/scripts/fixtures/tool-output/jscpd.json | 2 +- plugins/code-metrics/skills/audit-duplication/SKILL.md | 2 +- .../skills/audit-duplication/scripts/cluster-clones.py | 0 .../skills/audit-duplication/scripts/test_cluster_clones.py | 0 6 files changed, 4 insertions(+), 4 deletions(-) mode change 100644 => 100755 plugins/code-metrics/skills/audit-duplication/scripts/cluster-clones.py mode change 100644 => 100755 plugins/code-metrics/skills/audit-duplication/scripts/test_cluster_clones.py diff --git a/plugins/code-metrics/scripts/fixtures/tool-output/jscpd-aligned3.json b/plugins/code-metrics/scripts/fixtures/tool-output/jscpd-aligned3.json index 030fd71e1b..4af8a21c1b 100644 --- a/plugins/code-metrics/scripts/fixtures/tool-output/jscpd-aligned3.json +++ b/plugins/code-metrics/scripts/fixtures/tool-output/jscpd-aligned3.json @@ -106,4 +106,4 @@ "tokens": 330 } } -} \ No newline at end of file +} diff --git a/plugins/code-metrics/scripts/fixtures/tool-output/jscpd-offset3.json b/plugins/code-metrics/scripts/fixtures/tool-output/jscpd-offset3.json index 051ed487e1..aae97fbe47 100644 --- a/plugins/code-metrics/scripts/fixtures/tool-output/jscpd-offset3.json +++ b/plugins/code-metrics/scripts/fixtures/tool-output/jscpd-offset3.json @@ -106,4 +106,4 @@ "tokens": 314 } } -} \ No newline at end of file +} diff --git a/plugins/code-metrics/scripts/fixtures/tool-output/jscpd.json b/plugins/code-metrics/scripts/fixtures/tool-output/jscpd.json index f3171a6dbb..f1476b5721 100644 --- a/plugins/code-metrics/scripts/fixtures/tool-output/jscpd.json +++ b/plugins/code-metrics/scripts/fixtures/tool-output/jscpd.json @@ -68,4 +68,4 @@ "tokens": 220 } } -} \ No newline at end of file +} diff --git a/plugins/code-metrics/skills/audit-duplication/SKILL.md b/plugins/code-metrics/skills/audit-duplication/SKILL.md index 5f14e900d9..c76fa38bfb 100644 --- a/plugins/code-metrics/skills/audit-duplication/SKILL.md +++ b/plugins/code-metrics/skills/audit-duplication/SKILL.md @@ -1,5 +1,5 @@ --- -description: "Measure duplicated code as clone groups over the changed files, a path, or the whole tree: each group's duplicated lines and tokens with every instance's file and line range, per lane (TypeScript/JavaScript, Python, Bash, Go, C#) from whichever clone detector already resolves. Detector pairs are merged into clone classes so copies count once, and the report rolls classes up per lane and per directory. Replication the target repository declares about itself, a file vendored into several plugins and listed in a sanctioned-replication registry by path-within-plugin or as a canonical-to-copies cluster line, is subtracted from the total and reported as an exclusion naming the registry line rather than as debt, and the report emits no finding, no severity, and no exit-code gate. Use when: 'is this duplicated', 'find copy-paste code', 'clone detection', 'duplication report', 'how much of this change is copied', 'DRY check', 'redundant code', 'duplicated lines in the diff'; for lines per file use /code-metrics:audit-size, and for what a duplication number can and cannot support use /code-metrics:principles." +description: "Measure duplicated code as clone classes over the changed files, a path, or the whole tree: each class's duplicated lines and tokens with every instance's file and line range, rolled up per lane (TypeScript/JavaScript, Python, Bash, Go, C#) and per directory, from whichever clone detector already resolves. Replication the target repository declares about itself in a sanctioned-replication registry (a path-within-plugin or a canonical-to-copies cluster line) is subtracted from the total and reported as an exclusion naming the registry line rather than as debt; the report emits no finding, no severity, and no exit-code gate. Use when: 'is this duplicated', 'find copy-paste code', 'clone detection', 'duplication report', 'how much of this change is copied', 'DRY check', 'redundant code', 'duplicated lines in the diff'; for lines per file use /code-metrics:audit-size, and for what a duplication number can and cannot support use /code-metrics:principles." argument-hint: "[--json] [--all] [--base ] [--registry ] [...]" user-invocable: true disable-model-invocation: false diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/cluster-clones.py b/plugins/code-metrics/skills/audit-duplication/scripts/cluster-clones.py old mode 100644 new mode 100755 diff --git a/plugins/code-metrics/skills/audit-duplication/scripts/test_cluster_clones.py b/plugins/code-metrics/skills/audit-duplication/scripts/test_cluster_clones.py old mode 100644 new mode 100755 From a94d6f026db98311bb9f87870354e831ea1caa36 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 12 Sep 2026 03:37:27 +0000 Subject: [PATCH 15/15] fix(code-metrics): keep the no-detector headline when the other lane is not-applicable, defer detector facts to collectors.md The renderer's all-unavailable check counted the `other` lane's `not-applicable` duplication row as a detector row, so a whole-tree run with one file outside the language lanes and no detector on PATH never printed the consolidated install headline. Rows with that status are not probes and are left out of the check; two tests cover the mixed case and the only-not-applicable case. audit-duplication's SKILL.md restated which jscpd flags each major honours, what they default to, and which options dupl and CPD lack, with no basis, as-of date, or recheck trigger. Those are tool facts that move with the tools; the skill now states only what the adapters do and points at the duplication rows of reference/collectors.md, which carry each claim as a four-part verification record. The tokenization gotcha is deferred the same way. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01QShQwS5yDYYToBmofdEY5N --- plugins/code-metrics/scripts/report.py | 6 ++- plugins/code-metrics/scripts/test_report.py | 51 +++++++++++++++++++ .../skills/audit-duplication/SKILL.md | 15 +++--- 3 files changed, 65 insertions(+), 7 deletions(-) diff --git a/plugins/code-metrics/scripts/report.py b/plugins/code-metrics/scripts/report.py index dab6c2bc36..26159cc46b 100755 --- a/plugins/code-metrics/scripts/report.py +++ b/plugins/code-metrics/scripts/report.py @@ -451,8 +451,12 @@ def render( duplication = _is_duplication(doc) lines.append(f"# code-metrics: {doc.get('skill', '?')}") lines.append("") + # A `not-applicable` row (the `other` lane, which no detector covers) is + # not a probe that failed, so it neither earns the headline nor blocks it. detector_rows = [ - row for row in doc.get("run", []) if row.get("measure") == "duplication" + row + for row in doc.get("run", []) + if row.get("measure") == "duplication" and row.get("status") != "not-applicable" ] if ( duplication diff --git a/plugins/code-metrics/scripts/test_report.py b/plugins/code-metrics/scripts/test_report.py index f96e317261..8c94189190 100755 --- a/plugins/code-metrics/scripts/test_report.py +++ b/plugins/code-metrics/scripts/test_report.py @@ -1162,6 +1162,57 @@ def test_no_detector_prints_one_headline_with_the_hint(self) -> None: self.assertEqual(out.count("| unavailable | jscpd: not on PATH |"), 2) self.assertLess(out.index("No clone detector"), out.index("## Coverage")) + def test_a_not_applicable_lane_does_not_hide_the_no_detector_headline( + self, + ) -> None: + # The `other` lane carries a `not-applicable` duplication row on every + # run that has a file outside the language lanes; it is not a probe + # that failed, so it must not defeat the all-unavailable check. + hint = "jscpd: https://github.com/kucherenko/jscpd (npm install -g jscpd)" + doc = duplication_doc( + [], + status="empty", + run=[ + { + "lane": "bash", + "measure": "duplication", + "collector": None, + "status": "unavailable", + "reason": "jscpd: not on PATH", + "hint": hint, + }, + { + "lane": "other", + "measure": "duplication", + "collector": None, + "status": "not-applicable", + "reason": "no collector covers this lane", + "hint": None, + }, + ], + unavailable=["bash/duplication"], + ) + out = self.rendered(doc) + self.assertEqual(out.count("No clone detector ran in any lane"), 1) + self.assertEqual(out.count("npm install -g jscpd"), 1) + + def test_only_not_applicable_rows_print_no_headline(self) -> None: + doc = duplication_doc( + [], + status="empty", + run=[ + { + "lane": "other", + "measure": "duplication", + "collector": None, + "status": "not-applicable", + "reason": "no collector covers this lane", + "hint": None, + } + ], + ) + self.assertNotIn("No clone detector ran", self.rendered(doc)) + def test_a_lane_that_skipped_every_file_is_partial_not_empty(self) -> None: with tempfile.TemporaryDirectory() as tmp: d = Path(tmp) diff --git a/plugins/code-metrics/skills/audit-duplication/SKILL.md b/plugins/code-metrics/skills/audit-duplication/SKILL.md index c76fa38bfb..3297681a86 100644 --- a/plugins/code-metrics/skills/audit-duplication/SKILL.md +++ b/plugins/code-metrics/skills/audit-duplication/SKILL.md @@ -121,10 +121,12 @@ This script exports the five tunables to the collector adapters as `CODE_METRICS_DUP_MIN_TOKENS`, `CODE_METRICS_DUP_MIN_LINES`, `CODE_METRICS_DUP_IGNORE`, `CODE_METRICS_DUP_MAX_LINES`, and `CODE_METRICS_DUP_MAX_SIZE`, which is the only channel an adapter reads them through. `jscpd` passes the first three to the tool and applies the two caps -itself before the tool runs, because jscpd 4 and 5 disagree on what their own `--max-size` and -`--max-lines` default to and neither names a file it skipped; `dupl` and `cpd` have no -minimum-lines, ignore-glob, or cap option, so their adapters apply the minimum after parsing, -report the ignore globs as unused, and scan every file in scope. +itself before the tool runs; the `dupl` and `cpd` adapters apply the minimum after parsing, report +the ignore globs as unused, and scan every file in scope. Which options each tool honours, what its +own defaults are, and why the caps are applied here rather than passed through are tool facts that +move with the tools, so they are not restated here: the duplication rows of +`${CLAUDE_PLUGIN_ROOT}/reference/collectors.md` carry each claim with its basis, the date it was +verified, and the upstream event that obliges a recheck. `cpd` (PMD) sits after `jscpd` on `${CLAUDE_PLUGIN_ROOT}/scripts/collector-ladder.tsv` for every lane but Bash, so it runs only when `jscpd` does not resolve and `pmd` does. A repository that @@ -162,7 +164,8 @@ overrides are validated against the ladder file and an unknown name is dropped w - The pair merge is exact only for byte-identical copies. A class whose copies drifted by a line is reported as the detector saw it: the identical span as one class, and the drifted copy's shorter overlap as a second group naming the same file with a different range. -- jscpd 4 and jscpd 5 tokenize differently, so the same tree yields different class counts under - the two majors; compare runs made with one detector version, never across the boundary. +- Two versions of one detector can tokenize the same tree differently and so report different + class counts (the jscpd rows of `${CLAUDE_PLUGIN_ROOT}/reference/collectors.md` record the + verified case); compare runs made with one detector version, never across a version boundary. - Lowering `duplication.min_tokens` finds more and smaller clones, most of them boilerplate the language forces; the defaults are the detector's own conservative pair.