fix(cli): band evidence on the runs a finding rests on, not the runs its test appeared in

docs/cli/command-reference.md

@@ -110,6 +110,8 @@ Nothing inside the fence exceeds 72 columns, so it survives a phone and a quoted
 
 ### The population marker
 
+**The evidence level counts the runs behind the finding, not the runs behind the test.** `evidence moderate` means the claim was computed from somewhere between 8 and 15 independent runs — a `time-sensitive` split reads only the runs whose machine recorded a clock, and a `slower` finding only the runs whose durations could be normalised. So one test can carry two findings at two different levels, and that is the two claims resting on different amounts of data rather than an inconsistency. The exact number is `evidenceSessions` in the JSON. Whether a finding is shown at all is a separate bar, and that one does read the test's whole history in the window.
+
 The marker between the evidence level and the finding id — `all runs`, `-env` or `-env-cluster` — says **which runs went into the denominator** of that finding's rate. Every finding carries one, including the ones that set nothing aside, so that "we counted everything" and "this build did not say" never read alike.
 
 | Marker | The rate is over |
@@ -335,11 +337,11 @@ For scripts and agents. Emits a versioned envelope and nothing else — no rende
 xping report --all --format json > findings.json
 ```
 
-Every finding carries a `headline` — the same sentence the rendered report prints — plus `metrics`, the labelled pairs behind it, and the raw `evidence` the two were resolved from. It also carries `population`, which is one of `allExecutions`, `excludesEnvironmental` or `excludesEnvironmentalAndClustered` and says which executions the counts inside `evidence` were taken over:
+Every finding carries a `headline` — the same sentence the rendered report prints — plus `metrics`, the labelled pairs behind it, and the raw `evidence` the two were resolved from. It also carries `population`, which is one of `allExecutions`, `excludesEnvironmental` or `excludesEnvironmentalAndClustered` and says which executions the counts inside `evidence` were taken over, and `evidenceSessions`, the number `evidenceLevel` was banded from:
 
 ```json
 {
-  "schemaVersion": "1.13",
+  "schemaVersion": "1.14",
   "window": { "sessionCount": 20, "resolution": "default", "currentSliceSize": 3 },
   "context": { "sha": "a3f9c2e", "branch": "main", "assembly": "Checkout.Tests" },
   "summary": {
@@ -357,6 +359,7 @@ Every finding carries a `headline` — the same sentence the rendered report pri
       "kind": "Flaky",
       "severity": "high",
       "evidenceLevel": "moderate",
+      "evidenceSessions": 12,
       "population": "excludesEnvironmentalAndClustered",
       "subject": { "type": "test", "fullyQualifiedName": "…", "assembly": "Checkout.Tests",
                     "sourceFile": "tests/Billing/SummaryTests.cs", "sourceLineNumber": 88 },

docs/getting-started/local-first.md

@@ -108,7 +108,7 @@ the claim.
 | `HIGH` / `MED` / `LOW` | Impact ranking. Findings are ordered most severe first, so the top of the block is the part worth reading. |
 | `flaky`, `slower`, `stopped running` | The *kind* — what the finding claims. There are sixteen; the [CLI reference](../cli/command-reference.md#finding-kinds) lists them all. |
 | The counts line | The measurement the claim rests on, in plain numbers. |
-| `evidence low\|moderate\|high` | How much history stands behind it. A `low`-evidence finding is a lead, not a verdict. |
+| `evidence low\|moderate\|high` | How many runs stand behind *this claim* — which is not always every run the test appeared in, since a finding can only count the runs it could measure. A `low`-evidence finding is a lead, not a verdict. |
 | `f_2a91c0de` | A stable id for that finding, so you can refer to it in a ticket or diff two reports. |
 
 The report body sits inside a fenced code block and stays under 72 columns, so pasting it into

docs/internals/finding-populations.md

@@ -86,6 +86,42 @@ double-count. This is the arithmetic the field names exist to make possible: a t
 executions, ten clustered failures and two of its own publishes `2 of 12`, and `12` is not how many
 times it ran.
 
+## What each kind's evidence level counts
+
+The `evidence low|moderate|high` label on a finding is banded on the **runs that finding was
+computed from** — not on the runs its test appeared in. They are different numbers for every kind in
+the table above, and for the same reasons: a discounted run is not an occasion the test's own
+behaviour was observed on, and a run the kind could not read at all is not an occasion either.
+
+| kind | runs the level counts |
+|---|---|
+| `Flaky`, `AlwaysFailing`, `TimingOut` | runs of the test, less the environmental ones |
+| `RetryMasked` | the same |
+| `RetryDeepening` | the settled runs in both arms of the comparison |
+| `RetryExhausted` | `runsConsidered` |
+| `SharedFailure`, `BrokenFixture` | runs the cluster's best-evidenced member ran in — nothing is set aside |
+| `DurationRegression` | `current.comparedSessions` + `baseline.comparedSessions` |
+| `DurationUnstable` | the runs behind the normalised readings the dispersion was taken over |
+| `ParallelSensitive` | `trend.sessions` |
+| `TimeSensitive` | `worse.sessions` + `other.sessions` |
+| `Vanished` | `baselineSessions` — the habit the absence is a change from |
+
+Wherever the kind already publishes that figure, the level is banded on the published one, so the
+label and the counts a reader can check it against cannot drift apart. The number itself reaches the
+JSON as `evidenceSessions` on every finding, beside `evidenceLevel`.
+
+**Two findings about one test may therefore carry different levels.** That is the same statement the
+population marker already makes, one layer up: the kinds do not count the same runs, so they do not
+have the same amount of evidence either. A `TimeSensitive` split over the ten runs that recorded a
+clock is not better evidenced because the test also ran in ten that did not.
+
+**What is *not* banded this way is whether the finding is reported at all.** The reporting floor —
+`MinimumSessionsPerTestToReport`, applied in `FindingCoordinator` — reads the runs the subject
+appeared in, for every kind alike. Emission has to be one rule, or a test flagged by one metric is
+silently dropped by another with nothing on screen to explain it. So a claim resting on two runs of
+a test with twenty runs of history is still printed. It is printed saying `evidence low`, which is
+what the level is for.
+
 ## Why the exceptions are exceptions
 
 **`SharedFailure` and `BrokenFixture` keep environmental sessions.** An environmental session *is* a

src/Xping.Cli/Report/Contract/EnvelopeBuilder.cs

@@ -115,6 +115,7 @@ private static FindingDto BuildFinding(Finding finding)
             finding.Kind.ToString(),
             ToCamelCase(finding.Severity.ToString()),
             ToCamelCase(finding.EvidenceLevel.ToString()),
+            finding.EvidenceSessions,
             ToCamelCase(PopulationRules.For(finding.Kind).ToString()),
             BuildSubject(finding.Subject),
             headline,

src/Xping.Cli/Report/Contract/ReportEnvelope.cs

@@ -39,12 +39,11 @@ internal sealed record ReportEnvelope(
     /// Moves whenever anything a consumer reads changes shape, and the per-kind evidence payloads
     /// are part of that even though this document describes them as opaque: a script that reached
     /// into <c>evidence</c> for a field this build no longer emits is reading a contract, and
-    /// leaving the number still would tell it nothing had moved. 1.13 is where the summary gained
-    /// <c>partialSessions</c> and <c>VanishedEvidence</c> gained <c>partialSessionsSetAside</c> —
-    /// the runs that covered only part of the suite, which an absence is counted on neither side
-    /// of.
+    /// leaving the number still would tell it nothing had moved. 1.14 is where every finding gained
+    /// <c>evidenceSessions</c> — the denominator <c>evidenceLevel</c> is banded from, which is the
+    /// runs the finding was computed over rather than the runs its test appeared in.
     /// </remarks>
-    public const string CurrentSchemaVersion = "1.13";
+    public const string CurrentSchemaVersion = "1.14";
 }
 
 /// <summary>
@@ -122,6 +121,11 @@ internal sealed record SummaryDto(
 /// <param name="Kind">What the finding claims.</param>
 /// <param name="Severity">How much attention it deserves.</param>
 /// <param name="EvidenceLevel">How much data it rests on.</param>
+/// <param name="EvidenceSessions">
+/// Independent runs this finding was computed from — what <paramref name="EvidenceLevel"/> was
+/// banded from. Not the window's run count, and not the runs the test appeared in: a kind that
+/// discounts environmental runs, or that cannot read a run at all, measured over fewer.
+/// </param>
 /// <param name="Population">
 /// Which executions the counts and rates below were taken over. The report ranks findings of
 /// different kinds against each other and the kinds do not all count the same population, so two
@@ -137,6 +141,7 @@ internal sealed record FindingDto(
     string Kind,
     string Severity,
     string EvidenceLevel,
+    int EvidenceSessions,
     string Population,
     SubjectDto Subject,
     string Headline,

src/Xping.Cli/Report/FindingCoordinator.cs

@@ -66,7 +66,7 @@ public AnalysisResult Run(
         // time. Whether a p-value is worth reporting depends on how many fingerprints the kind was
         // tested on and on what the other survivors read, neither of which is known until every
         // provider has run.
-        var surviving = new List<(FindingCandidate Candidate, int Sessions)>();
+        var surviving = new List<FindingCandidate>();
         var tested = new Dictionary<FindingKind, int>();
 
         int lowEvidence = 0;
@@ -107,15 +107,22 @@ public AnalysisResult Run(
                 if (kinds != null && !kinds.Contains(candidate.Kind))
                     continue;
 
-                int sessions = EvidenceLevelResolver.CountSessions(candidate.Subject, context.Tests);
+                // The subject's own history, not the candidate's. The floor asks whether this test
+                // has been around long enough to be judged at all, which is a property of the test
+                // and has to be answered the same way for every kind — otherwise one metric flags a
+                // test that another silently drops, with nothing on screen to explain it. What the
+                // candidate measured over decides its evidence band further down, and only that.
+                int subjectSessions =
+                    EvidenceLevelResolver.CountSessions(candidate.Subject, context.Tests);
 
-                if (!EvidenceLevelResolver.MeetsReportingFloor(sessions, context.Window.SessionCount))
+                if (!EvidenceLevelResolver.MeetsReportingFloor(
+                        subjectSessions, context.Window.SessionCount))
                 {
                     lowEvidence++;
                     continue;
                 }
 
-                surviving.Add((candidate, sessions));
+                surviving.Add(candidate);
             }
         }
 
@@ -124,7 +131,7 @@ public AnalysisResult Run(
         var findings = new List<Finding>();
         int notSignificant = 0;
 
-        foreach ((FindingCandidate collected, int sessions) in surviving)
+        foreach (FindingCandidate collected in surviving)
         {
             FindingCandidate? candidate = Reported(collected, cutoffs);
 
@@ -140,7 +147,12 @@ public AnalysisResult Run(
                 FindingId.Compute(candidate.Kind, candidate.Subject.SortKey),
                 candidate.Kind,
                 candidate.Cap(ImpactScorer.Band(impact)),
-                EvidenceLevelResolver.Resolve(sessions),
+
+                // Read off the candidate that is actually being reported, so an `Instead` handover
+                // is banded on what the replacement claim was measured over rather than on what the
+                // claim it replaced was.
+                EvidenceLevelResolver.Resolve(candidate.EvidenceSessions),
+                candidate.EvidenceSessions,
                 candidate.Subject,
                 candidate.Evidence,
                 candidate.DrillDownCommand,
@@ -175,12 +187,12 @@ public AnalysisResult Run(
     /// </para>
     /// </remarks>
     private static Dictionary<FindingKind, double?> Cutoffs(
-        IReadOnlyList<(FindingCandidate Candidate, int Sessions)> surviving,
+        IReadOnlyList<FindingCandidate> surviving,
         IReadOnlyDictionary<FindingKind, int> tested)
     {
         var byKind = new Dictionary<FindingKind, List<double>>();
 
-        foreach ((FindingCandidate candidate, _) in surviving)
+        foreach (FindingCandidate candidate in surviving)
         {
             if (candidate.PValue is not { } p)
                 continue;

src/Xping.Cli/Report/LocalAnalysisConstants.cs

@@ -42,6 +42,13 @@ internal static class LocalAnalysisConstants
     /// session that retried five times cleared an execution-denominated floor of five on its own —
     /// exactly the shape the floor exists to exclude, and worst for the tests that retry most.
     /// </para>
+    /// <para>
+    /// The subject's sessions, deliberately, and not the ones a given finding was computed from —
+    /// which is what <see cref="EvidenceModerateSessions"/> bands. Whether a test has enough
+    /// history to be judged has to be answered once for every kind, or one metric flags a test that
+    /// another silently drops with nothing on screen to explain the difference. A claim resting on
+    /// less than its subject's history is still reported; it is reported saying <c>low</c>.
+    /// </para>
     /// </remarks>
     public const int MinimumSessionsPerTestToReport = 5;
 
@@ -430,10 +437,16 @@ internal static class LocalAnalysisConstants
     public const double SeverityMediumThreshold = 0.30;
 
     /// <summary>
-    /// Sessions a test must have run in before its evidence is better than <c>Low</c> (8).
+    /// Runs a finding must rest on before its evidence is better than <c>Low</c> (8).
     /// </summary>
     /// <remarks>
     /// <para>
+    /// The runs the finding was computed from, not the runs its test appeared in — see
+    /// <see cref="Scoring.EvidenceLevelResolver"/> for why those are different numbers and why only
+    /// this one is banded. It brings the CLI closer to Cloud rather than further from it: Cloud's
+    /// effective sample size is computed over the rows its own analysis read.
+    /// </para>
+    /// <para>
     /// Banded on sessions, which is the unit Xping Cloud already bands on: its
     /// <c>EvidenceLevelThresholds</c> classifies an effective sample size computed over runs that
     /// <c>RunCollapser</c> has reduced to one row per test per session. The unit agrees; the numbers
@@ -449,7 +462,7 @@ internal static class LocalAnalysisConstants
     public const int EvidenceModerateSessions = 8;
 
     /// <summary>
-    /// Sessions above which evidence is <c>High</c> (15).
+    /// Runs above which evidence is <c>High</c> (15).
     /// </summary>
     public const int EvidenceHighSessions = 15;
 

src/Xping.Cli/Report/Model/Finding.cs

@@ -92,7 +92,14 @@ internal abstract record FindingEvidence;
 /// <param name="Id">Stable short identity; see <see cref="FindingId"/>.</param>
 /// <param name="Kind">What the finding claims.</param>
 /// <param name="Severity">Banded from <paramref name="Impact"/>.</param>
-/// <param name="EvidenceLevel">How much data the claim rests on.</param>
+/// <param name="EvidenceLevel">How much data the claim rests on, banded.</param>
+/// <param name="EvidenceSessions">
+/// The number <paramref name="EvidenceLevel"/> was banded from: independent runs the claim was
+/// computed from, which is generally fewer than the runs the subject appeared in. Carried and
+/// published alongside the band for the reason every other threshold in the report publishes its
+/// input — a level a reader cannot reconcile with the counts beside it is a level they have to take
+/// on trust.
+/// </param>
 /// <param name="Subject">The test or group the finding is about.</param>
 /// <param name="Evidence">The kind-specific payload.</param>
 /// <param name="DrillDownCommand">
@@ -108,6 +115,7 @@ internal sealed record Finding(
     FindingKind Kind,
     Severity Severity,
     EvidenceLevel EvidenceLevel,
+    int EvidenceSessions,
     FindingSubject Subject,
     FindingEvidence Evidence,
     string DrillDownCommand,

src/Xping.Cli/Report/Model/Severity.cs

@@ -26,7 +26,7 @@ internal enum Severity
 }
 
 /// <summary>
-/// How much data a finding rests on, measured in sessions the subject ran in within the window.
+/// How much data a finding rests on, measured in the runs it was computed from.
 /// </summary>
 /// <remarks>
 /// <para>
@@ -35,6 +35,14 @@ internal enum Severity
 /// must not be labelled as though it had forty.
 /// </para>
 /// <para>
+/// The finding's runs, not the subject's. Every kind measures over a subset of the runs its test
+/// appeared in — environmental runs are discounted, and a run whose session recorded no clock or
+/// whose durations cannot be normalised is one the question could not be asked of. The number is
+/// carried on the finding as <c>EvidenceSessions</c> and published beside the level, so a reader
+/// can see which of the two it came from. Whether a finding is emitted at all is a separate
+/// judgement, made on the subject — see <see cref="Scoring.EvidenceLevelResolver"/>.
+/// </para>
+/// <para>
 /// The unit matches Xping Cloud, which bands an effective sample size computed over runs collapsed
 /// to one row per test per session. The thresholds do not, and deliberately —
 /// <see cref="LocalAnalysisConstants.EvidenceModerateSessions"/> explains why. Local windows are