fix(cli): band evidence on the runs a finding rests on, not the runs its test appeared in - #206
Merged
Merged
Conversation
…its test appeared in EvidenceLevelResolver.CountSessions read TestIndex.SessionsRunIn — every session the subject ran in — and FindingCoordinator banded the printed evidence level on that. No provider computes its claim from all of them. ClocksIn drops every session that recorded no UtcOffset and every environmental one; ParallelSensitive drops executions with no orchestration record; a duration reading can only be normalised when its own run recorded a positive median; the failure-mode and retry kinds discount environmental runs; Vanished sets aside the runs that covered only part of the suite. So a test present in all twenty window sessions but readable in ten published a split computed from ten runs as `evidence high`. The mismatch predates the 8/15 bands; those bands brought it into the top one. FindingCandidate now carries EvidenceSessions, required rather than defaulted so the next kind added cannot inherit this in silence, and the coordinator bands on it. Where a kind already publishes the figure — worse.sessions + other.sessions, trend.sessions, comparedSessions, baselineSessions — the same value is handed over, so the level and the counts a reader can check it against cannot drift. It is read off the candidate that is actually reported, so an Instead handover bands on its own denominator rather than on that of the claim it replaced. Finding and the JSON envelope publish it as evidenceSessions; the schema goes to 1.14. The reporting floor deliberately did not move. EvidenceLevelResolver's class remark defends against a test being confidently flagged by one metric and silently dropped by another, and that hazard is about emission: MinimumSessionsPerTestToReport still reads the subject's whole history, for every kind alike, and nothing that was reported before stops being reported. What moves is the label, which is a description of one claim rather than a gate on it — so two findings about one test may now carry different levels, exactly as they already carry different population markers. The trade is recorded in the class remark and in docs/internals/finding-populations.md, which gains the per-kind table. Clustered failures do not shorten the count: they remove a failure, not a run, which is what the population doc already says of sessionsConsidered. Findings over the sample store are byte-identical to main but for the new field. Closes #182 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TMQcZLBGFTd8wz3mcfn9vc
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The implementation consistently applies claim-specific denominators and includes comprehensive regression and contract coverage.
Pull request overview
Corrects CLI evidence levels to reflect the independent runs supporting each finding while preserving the subject-based reporting floor.
Changes:
- Adds and publishes
EvidenceSessions, with schema version 1.14. - Updates every provider and alternative handover to supply its actual denominator.
- Adds provider, coordinator, contract, and documentation coverage.
File summaries
| File | Description |
|---|---|
docs/cli/command-reference.md |
Documents evidence-session semantics and JSON field. |
docs/getting-started/local-first.md |
Clarifies rendered evidence levels. |
docs/internals/finding-populations.md |
Defines denominators for every finding kind. |
src/Xping.Cli/Report/Contract/EnvelopeBuilder.cs |
Maps evidence sessions into JSON. |
src/Xping.Cli/Report/Contract/ReportEnvelope.cs |
Adds the field and bumps schema to 1.14. |
src/Xping.Cli/Report/FindingCoordinator.cs |
Separates reporting-floor and banding denominators. |
src/Xping.Cli/Report/LocalAnalysisConstants.cs |
Clarifies threshold semantics. |
src/Xping.Cli/Report/Model/Finding.cs |
Stores the evidence-session count. |
src/Xping.Cli/Report/Model/Severity.cs |
Updates evidence-level documentation. |
src/Xping.Cli/Report/Providers/DurationProvider.cs |
Counts comparable or normalizable runs. |
src/Xping.Cli/Report/Providers/FailureModeProvider.cs |
Counts non-environmental observation runs. |
src/Xping.Cli/Report/Providers/IFindingProvider.cs |
Requires provider-supplied evidence sessions. |
src/Xping.Cli/Report/Providers/ParallelSensitiveProvider.cs |
Uses sessions carrying concurrency data. |
src/Xping.Cli/Report/Providers/RetryProvider.cs |
Supplies kind-specific retry denominators. |
src/Xping.Cli/Report/Providers/TimeSensitiveProvider.cs |
Uses runs carrying clock data. |
src/Xping.Cli/Report/Providers/VanishedProvider.cs |
Uses baseline appearances. |
src/Xping.Cli/Report/Scoring/EvidenceLevelResolver.cs |
Bands candidate evidence independently of emission. |
tests/Xping.Cli.Tests/Commands/CliSurfaceTests.cs |
Updates schema assertions. |
tests/Xping.Cli.Tests/Report/DurationProviderTests.cs |
Covers duration denominators. |
tests/Xping.Cli.Tests/Report/ExitCodeTests.cs |
Updates finding fixtures. |
tests/Xping.Cli.Tests/Report/FailureModeProviderTests.cs |
Covers environmental and clustered runs. |
tests/Xping.Cli.Tests/Report/FindingCoordinatorTests.cs |
Covers banding, floors, and alternatives. |
tests/Xping.Cli.Tests/Report/FindingOrderTests.cs |
Updates finding fixtures. |
tests/Xping.Cli.Tests/Report/ParallelSensitiveProviderTests.cs |
Covers concurrency-readable runs. |
tests/Xping.Cli.Tests/Report/ReportEnvelopeTests.cs |
Verifies schema and published denominator. |
tests/Xping.Cli.Tests/Report/RetryProviderTests.cs |
Covers retry-kind denominators. |
tests/Xping.Cli.Tests/Report/ShareableOutputTests.cs |
Updates envelope fixtures. |
tests/Xping.Cli.Tests/Report/TimeSensitiveProviderTests.cs |
Covers clock-readable runs. |
tests/Xping.Cli.Tests/Report/VanishedProviderTests.cs |
Covers baseline-only evidence. |
Review details
- Files reviewed: 29/29 changed files
- Comments generated: 0
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #182.
The problem
EvidenceLevelResolver.CountSessionsreadTestIndex.SessionsRunIn— every session the subject ran in — andFindingCoordinatorbanded the printed evidence level on that. No provider computes its claim from all of them:TimeSensitiveUtcOffsetParallelSensitiveDurationMaxComparedSessionsFailureModeRetryVanishedSo a test present in all twenty window sessions but readable in ten published a split computed from ten runs as
evidence high. The mismatch predates the 8/15 bands from #179; those bands brought it into the top one.The fix
FindingCandidatecarriesEvidenceSessions— the independent runs the claim was computed from — and the coordinator bands on it. Required rather than defaulted: a default would be some other number, and the next kind added would inherit this in silence. The compiler found all twelve construction sites.Where a kind already publishes the figure, the same value is handed over rather than recomputed, so the level and the counts a reader can check it against cannot drift apart:
Flaky,AlwaysFailing,TimingOut,RetryMaskedRetryDeepeningRetryExhaustedrunsConsideredSharedFailure,BrokenFixtureDurationRegressioncurrent.comparedSessions+baseline.comparedSessionsDurationUnstableParallelSensitivetrend.sessionsTimeSensitiveworse.sessions+other.sessionsVanishedbaselineSessionsIt is read off the candidate that is actually reported, so an
Insteadhandover bands on its own denominator rather than on that of the claim it replaced —DurationUnstablereads what could be normalised where theDurationRegressionit stands in for read what could be compared.Clustered failures do not shorten the count. They remove a failure, not a run, which is what
docs/internals/finding-populations.mdalready says ofsessionsConsidered.Why the class remark survives it
#182 notes that
EvidenceLevelResolver's remark makes evidence a property of the subject so that "two providers disagreeing about what counts as enough evidence" cannot produce a report where a test is confidently flagged by one metric and silently dropped by another — and asks whichever fix is taken to say why the trade is worth it.The hazard it names is emission, and emission did not move.
MinimumSessionsPerTestToReportstill reads the subject's whole history, in one function, for every kind alike; nothing that was reported before stops being reported. A claim resting on two runs of a test with twenty runs of history is still printed — it is printed sayingevidence low, which is what the level is for.What moved is the label, which is a description of one claim rather than a gate on it. So two findings about one test may now carry different levels, exactly as they already carry different population markers: the kinds do not count the same runs, so they do not have the same amount of evidence either. This also brings the CLI closer to Cloud rather than further from it — Cloud's
EvidenceLevelThresholdsbands an ESS computed over the rows its own analysis read.The trade is recorded in the class remark, on
MinimumSessionsPerTestToReport, and in a new section ofdocs/internals/finding-populations.md.Published
Findingand the JSON envelope gainevidenceSessionsbesideevidenceLevel— the same pattern as #196, so a level a reader cannot reconcile with the counts beside it is not something they have to take on trust. Schema goes to 1.14. The rendered text report is unchanged; 72 columns are already spent.Verification
Insteadhandover) and one per provider. TheTimeSensitiveone is the issue's own example — twenty runs, ten carrying a clock — and fails onmain.--assembly SampleApp.XUnit, 20 runs), findings are byte-identical tomainapart from the new field and the schema version. That store holds no environmental runs, so all four denominators are 20 — a confirmation that nothing moved sideways rather than a demonstration of the fix.🤖 Generated with Claude Code
https://claude.ai/code/session_01TMQcZLBGFTd8wz3mcfn9vc