fix(cli): publish the spread the duration comparison is not calibrated against

[xping-admin]

BrunnerMunzel.OneSidedPValue calibrates by relabelling the pooled readings — every way the runs could have been split between the two arms, with the observed one counted. That is exact under exchangeability: both arms drawn from one distribution.

Brunner–Munzel's own null is weaker. It asks only that neither arm be stochastically larger and explicitly permits the two to differ in spread, which is the reason #168 chose it over Mann–Whitney in the first place. Studentising is what carries a permutation calibration to that weaker null, and it does so asymptotically. Three recent runs are not asymptotic.

So RegressionAlpha is a ceiling in one direction only. Both arms centred so stochastic superiority is exactly one half — nothing has slowed — seventeen readings against three, read at 0.01:

σ baseline	σ recent	p ≤ 0.01	whole gate chain
0.8	0.2	.0002	.0002
0.2	0.2	.0100	.0099
0.2	0.4	.0386	.0203
0.2	0.8	.0729	.0653

The conservative direction is the common one — a fortnight of history usually spreads wider than three runs — but the liberal one is real.

#186 corrected the claim and pinned the residual with two theories. This does the other half of #187's acceptance: the residual is published where a reader meets it.

Why this direction and not the other three

The issue lists four. Taking them in turn, because three of them are worse than they look:

A statistic less sensitive to the spread ratio is a dead end, and it is worth saying why rather than leaving the next reader to re-derive it. The rejections come from complete separation — all three wild recent readings landing above a tight baseline, which at σ 0.2/0.8 happens on the order of 3% of draws under the null. There, the recent pooled ranks are 18, 19, 20 and their within-arm ranks 1, 2, 3, so both rank variances are exactly zero, the statistic is +∞, and the p-value is the floor. Every rank test returns its floor on that arrangement; ranks are all any of them can see. The unstudentised Mann–Whitney is strictly worse — it rejects on all such draws, where studentising at least declines some of them — which is #168's argument arriving from the other end.

Widening the recent slice attacks the real cause and is the only lever that would move the rate, but it is its own change: MaxComparedSessions has to fall from 40 to about 15 to keep C(n + k, k) under TheEnumerationTheExactTestPerformsStaysBounded's bar, MinimumBaselineSessions has to be re-derived from the new floor, and CurrentSliceSize is the report's shared notion of "now" — so it either moves for every delta kind or duration stops meaning by "recent" what its neighbours mean. Left for a follow-up rather than folded in here.

Rerouting to DurationUnstable where the spreads disagree reads well and does not survive contact with the data. Conditioning on all-three-above truncates the recent arm from below, which compresses exactly the spread such a guard would read, so it catches roughly half the cases while costing power on genuinely noisy real regressions. The issue already rejects its close cousin — a dispersion guard on a three-reading arm — for a related reason.

That leaves publishing, which is what this does.

What it publishes

Both arms of a DurationRegression now carry ComparedDispersion, beside the ComparedSessions it is computed over.

Measured by RobustDispersion, the same small-sample-corrected statistic DurationUnstable is decided on, whose median-unbiasing table starts at two readings — so it is defined at the three-run arm floor rather than needing a special case there.

Over Profile.Compared, the one-normalised-reading-per-run sample the two-sample test actually read, and deliberately not over Profile.Normalised. DurationUnstableEvidence.Dispersion is per-execution on purpose — an attempt that took longer than its neighbour is part of how much a test's timing moves, however correlated the two are — while a comparison's arms cannot be, because the test they feed assumes independence. A spread computed over the wrong one of those two samples would describe a comparison that was never made. TheSpreadPublishedBesideTheComparisonIsOverTheRunsItRead is the test that keeps them apart: seven baseline runs retrying at 100ms, 200ms and 1200ms publish an arm spread of exactly zero, while the instability finding the regression suppressed reads 4.1 over the same window.

Directly under significance in the finding's metrics, because it is a qualifier on that number rather than another fact about the test:

"metrics": [
  { "label": "significance", "value": "p 0.008333 one-sided, 3 recent runs against 7" },
  { "label": "spread",       "value": "recent 1.05 against baseline 0 over the compared runs" }
],
"evidence": {
  "current":  { "comparedSessions": 3, "comparedDispersion": 1.048 },
  "baseline": { "comparedSessions": 7, "comparedDispersion": 0 }
}

Worth being exact about where that is, because an earlier revision of this branch was not: it reaches a reader through --format json. The terminal report prints each finding's headline and nothing else — TextReportRenderer.WriteFinding never reads FindingDto.Metrics — and that is true of every kind, per ReportEnvelope's own note that metrics are "the same observations as labelled pairs, for a caller laying out its own". Whether a human reading xping report should also meet this pair is a real question and a separate one; it would mean the headline sentence, which is the only thing that renderer prints.

Relative to each arm's own median, which is what makes the two comparable rather than an inconsistency: every regression has a recent arm at least half again the baseline's level by the gate that admitted it, so an absolute spread would read larger on the slower arm for no reason but its being the slower arm. Durations are perturbed multiplicatively and the effect size beside this is a ratio — on that scale, two arms that vary alike read alike however far apart their levels are.

Two figures rather than their quotient, which is a deliberate departure from the issue's wording — it asks for the dispersion ratio. A steady baseline reads exactly zero, and a fast test quantised to the same millisecond in every run is the commonest shape in this data — it is the shape BrunnerMunzel's whole degenerate-case convention exists for. The ratio's denominator is therefore zero precisely where the comparison is most confident, and an infinity in the evidence is not a number a reader can weigh anything against. Two numbers they can divide themselves are strictly more informative and carry no convention.

Does publishing it actually help

Only if the spread really does mark the findings that shouldn't be there, so that is measured rather than assumed. AFalseSlowdownFromAWilderRecentSlicePublishesTheSpreadThatCausedIt re-runs the σ 0.2 / 0.8 cell — every finding it produces is false, both arms are centred on 200ms — and asks what the evidence beside those findings says. Over four thousand windows it produces 296 findings, 251 of which publish the recent arm as the wider one: 0.848.

The shortfall is honest rather than slack in the band. Three readings from a wide distribution can land close together, and when they do the finding looks exactly like a real step and nothing in the data says otherwise. That 15% is the part of the residual no amount of publishing reaches.

Documentation

docs/known-limitations.md has a ## Local Analysis section with an entry per kind and had none for DurationRegression. It now does: what the calibration is exact for, the table above in a reader's terms, why there is no fix, what the finding is really saying when it fires — the recent runs genuinely have changed, in dispersion rather than in location, and DurationUnstable is the kind that claim belongs to — and how to read the two published spreads. Version history moves to 1.11.0.

command-reference.md had per-kind prose for ParallelSensitive, TimeSensitive, the retry kinds, BrokenFixture and Vanished, and nothing for DurationRegression; it now has a paragraph in the same voice, pointing at the limitation. local-first.md's one-liner gains the same pointer.

Published shape

DurationProfile gains comparedDispersion on both arms, so the schema moves 1.15 → 1.16. Nothing leaves.

Verification

• 968/968 Xping.Cli.Tests pass, up from 964. 901/901 Xping.Sdk.Core.Tests.
• Four new tests, plus ComparedDispersion assertions folded into the existing evidence theory. AFindingWhoseRecentSliceIsTheWilderArmSaysSoInTheEvidence states its two spreads as hand-computed literals — 4, 8 and 16 about a median of 8 give a scaled deviation of 1.4826 × 4 against a scaled interquartile range of 6 / 1.349, the larger wins, and 5.9304 × 1.4136 / 8 = 1.048 — on a fixture whose p-value is identical to a clean step's, which is the reader's problem stated as a test.
• Both calibration theories pin the same bands they did on main. They must: nothing statistical moved, and a shifted band would mean something else had.
• The report over this repository's own local store differs from main's in exactly one line, the schema version. That is a weak statement and worth saying so: the store's largest suite holds thirteen runs and produces no duration finding of either kind.

Closes #187

🤖 Generated with Claude Code

https://claude.ai/code/session_01TMQcZLBGFTd8wz3mcfn9vc

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines	Coverage Δ
src/Xping.Cli/Report/Contract/EvidenceHeadline.cs	94.13% <100.00%> (+0.28%)	⬆️
src/Xping.Cli/Report/Contract/ReportEnvelope.cs	100.00% <ø> (ø)
src/Xping.Cli/Report/Providers/DurationProvider.cs	97.32% <100.00%> (+0.02%)	⬆️
src/Xping.Cli/Report/Scoring/BrunnerMunzel.cs	98.40% <ø> (ø)

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

🟡 Changes recommended

Documentation currently misstates the threshold and available mitigation and contains a duplicate separator.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Publishes per-arm dispersion for duration regressions so readers can assess permutation-calibration limitations.

Changes:

• Adds and renders ComparedDispersion.
• Bumps report schema to 1.16.
• Adds statistical tests and user documentation.

File summaries

File	Description
src/Xping.Cli/Report/Providers/DurationProvider.cs	Computes and publishes dispersion.
src/Xping.Cli/Report/Contract/EvidenceHeadline.cs	Renders both spreads.
src/Xping.Cli/Report/Contract/ReportEnvelope.cs	Bumps schema version.
src/Xping.Cli/Report/Scoring/BrunnerMunzel.cs	Documents calibration limitations.
tests/Xping.Cli.Tests/Report/DurationProviderTests.cs	Tests dispersion evidence.
tests/Xping.Cli.Tests/Report/BrunnerMunzelTests.cs	Clarifies calibration coverage.
tests/Xping.Cli.Tests/Report/ShareableOutputTests.cs	Updates evidence fixtures.
tests/Xping.Cli.Tests/Report/ReportEnvelopeTests.cs	Updates schema assertions.
tests/Xping.Cli.Tests/Commands/CliSurfaceTests.cs	Updates CLI schema assertions.
docs/known-limitations.md	Documents statistical limitations.
docs/getting-started/local-first.md	Links duration guidance.
docs/cli/command-reference.md	Documents duration comparison evidence.

Review details

Suppressed comments (1)

docs/known-limitations.md:446

• This adds a second horizontal rule immediately before the existing one at line 448, producing a duplicate separator before “General Limitations.” Keep only one rule.

---

• Files reviewed: 12/12 changed files
• Comments generated: 2
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[xping-admin]

All three addressed in 67ee1ef, including the suppressed one.

The duplicate separator (docs/known-limitations.md, suppressed comment) is real but is not this branch's. main already carries two consecutive rules before ## General Limitations:

$ git show main:docs/known-limitations.md | sed -n '395,401p'
exactly like one that needs one.

---

---

## General Limitations

My section was inserted above that pair and contributed exactly one rule of its own, in line with every other ### entry in the file. Removed the stray one anyway — it is a one-line tidy in a section this branch is already editing, and the rule count is now back to main's 11.

The other two are in the threads. Both were right: "the factor must clear a half" documented a 0.5x bar where the gate reads ratio - 1 >= 0.50, and "there is no fix available" collapsed two different claims — no exact correction exists for this comparison, but a wider recent slice is a real lever that this PR defers rather than one that does not exist.

968/968 Xping.Cli.Tests still pass; the change is prose only.

[xping-admin]

Ran /code-review medium over the branch. Four findings; two were right and land in 8a1225b, two I am pushing back on.

Fixed — the docs pointed at output the CLI never produces. The How to tell section told a reader to look for a spread row under the p-value and showed a terminal-style block containing one. No such line is printed. TextReportRenderer.WriteFinding writes the marker, the label, the subject, the wrapped headline and the dim trailer, and never reads FindingDto.Metrics — which is deliberate and true of every kind, per ReportEnvelope's own note that metrics are "the same observations as labelled pairs, for a caller laying out its own". So the fence is now the JSON it always was, the instruction names --format json, the same overstatement is gone from command-reference.md, and the two source comments that said "the rendered finding states the two side by side" say "the finding's metrics" instead. The JSON paths beside the block were already right. local-first.md's "the evidence states each side's spread" was accurate and is unchanged.

Worth naming what that leaves, since it is the honest limit of this PR: a human reading xping report meets the headline and nothing else, so this pair reaches them only through JSON. Whether the headline sentence should carry it is a real question and a separate change — it is the only string that renderer prints, and every kind's metrics are invisible for the same reason.

Fixed — the relative scale was undocumented. Raised as an inconsistency: RobustDispersion divides by each arm's own median, and every regression has a recent arm at least 1.5x the baseline's level by the gate that admitted it, so two arms with identical absolute spread publish different figures. That is the intended behaviour rather than a bug — durations are perturbed multiplicatively, the effect size beside this is a ratio, and on that scale two arms that vary alike read alike however far apart their levels are; an absolute spread would read larger on the slower arm for no reason but its being the slower arm. But a reader can make exactly that misreading, so the <param> remark and the limitations page now say it outright.

Not changed — publication precision. Flagged that FindingOrder.Round (3 dp) and Rate (0.##) collapse a baseline of 0.001 against a recent of 0.004 into "recent 0 against baseline 0", and that this is the false-positive shape. It is not. Spreads that small cannot produce a finding under the null: the practical gate demands a Hodges-Lehmann ratio of at least 1.5, which against a spread of 0.004 is a gap of some hundreds of times the arms' own variation — an enormous, clean, real shift, and precisely the regime where the calibration worry does not arise. DurationUnstableEvidence.Dispersion publishes at the same 3 dp against a threshold of 0.65, so the precision is also the one the neighbouring field already uses.

Not changed — no other issues. The review confirmed the parts worth confirming: RobustDispersion.Of(Compared) cannot divide by zero because non-positive readings are filtered out in Compared, both arms are floored at 3 and 7 readings so the < 2 degenerate path is unreachable, Compared.Count is capped at 40 and stays inside the correction table, and DurationProfile has no construction site outside ToPublished and the test fixture.

968/968 Xping.Cli.Tests pass. The change is prose only.