fix(cli): count the tests a metric could not measure, not only the candidates the report dropped (#185)

docs/cli/command-reference.md

@@ -337,11 +337,18 @@ For scripts and agents. Emits a versioned envelope and nothing else — no rende
 xping report --all --format json > findings.json
 ```
 
-Every finding carries a `headline` — the same sentence the rendered report prints — plus `metrics`, the labelled pairs behind it, and the raw `evidence` the two were resolved from. It also carries `population`, which is one of `allExecutions`, `excludesEnvironmental` or `excludesEnvironmentalAndClustered` and says which executions the counts inside `evidence` were taken over, and `evidenceSessions`, the number `evidenceLevel` was banded from:
+Every finding carries a `headline` — the same sentence the rendered report prints — plus `metrics`, the labelled pairs behind it, and the raw `evidence` the two were resolved from. It also carries `population`, which is one of `allExecutions`, `excludesEnvironmental` or `excludesEnvironmentalAndClustered` and says which executions the counts inside `evidence` were taken over, and `evidenceSessions`, the number `evidenceLevel` was banded from.
+
+`summary.notMeasured` says, per kind, how many tests that metric could not be computed for at all —
+split into the ones waiting for more runs and the ones whose recorded data cannot answer the question
+however long you wait. It is deliberately not a total: adding the entries counts a test once per
+question it could not answer, and a kind absent from the object keeps no such tally. Read one entry to
+ask "how much of my suite could this metric read". Tests counted here are inside `healthy`, which
+means "no finding was raised" rather than "checked and fine":
 
 ```json
 {
-  "schemaVersion": "1.14",
+  "schemaVersion": "1.15",
   "window": { "sessionCount": 20, "resolution": "default", "currentSliceSize": 3 },
   "context": { "sha": "a3f9c2e", "branch": "main", "assembly": "Checkout.Tests" },
   "summary": {
@@ -351,6 +358,10 @@ Every finding carries a `headline` — the same sentence the rendered report pri
     "healthy": 409,
     "excludedLowEvidence": 41,
     "excludedNotSignificant": 6,
+    "notMeasured": {
+      "DurationRegression": { "awaitingRuns": 63, "unreadable": 32 },
+      "ParallelSensitive": { "awaitingRuns": 0, "unreadable": 108 }
+    },
     "partialSessions": 0
   },
   "findings": [

docs/internals/finding-populations.md

@@ -122,6 +122,95 @@ silently dropped by another with nothing on screen to explain it. So a claim res
 a test with twenty runs of history is still printed. It is printed saying `evidence low`, which is
 what the level is for.
 
+## What each kind could not measure
+
+A finding's population says which executions its counts were taken over. This section says something
+prior to that: which **tests the kind could not be computed for at all**, and why. Until it was
+recorded, a provider that declined for want of data left no trace anywhere — the summary's excluded
+tally counts only the candidates the coordinator itself dropped at the reporting floor — so a test no
+statistic could be taken of fell through into `healthy` and was reported to a reader as fine.
+
+The tally reaches the JSON as `summary.notMeasured`, keyed by kind, and the rendered report as one
+line above the fence naming the three largest with `+n more`.
+
+### The two reasons, and the question that separates them
+
+| reason | meaning | what a reader does |
+|---|---|---|
+| `awaitingRuns` | the window does not hold enough of this test yet | wait; the store fills |
+| `unreadable` | nothing this kind reads was recorded | change what is recorded, or accept it |
+
+The question at every decline site is whether **another run of the same shape as the ones already
+recorded** would fix it. Nothing subtler is needed: the two differ in what the reader should do next,
+not in how the gate was written. Six baseline runs where the comparison needs seven is `awaitingRuns`;
+a test whose every run recorded a zero median normalises nothing, and the eighth such run normalises
+nothing either, so it is `unreadable`.
+
+**This is not the reporting floor.** That floor asks whether a test has been around long enough to be
+judged and is applied centrally, to a candidate a provider already computed. This counts tests for
+which no candidate was ever computed. A candidate that reaches the coordinator has been measured by
+definition, so the two can never both fire on one claim.
+
+### The matrix
+
+| kind | `awaitingRuns` | `unreadable` |
+|---|---|---|
+| `DurationRegression` | either arm under 7 / 3 comparable runs | an arm that held runs and normalised none of them |
+| `DurationUnstable` | one normalised reading | nothing normalisable, or no run with a usable median |
+| `ParallelSensitive` | structurally none — this kind has no session floor | fewer than two distinct concurrency levels |
+| `TimeSensitive` | fewer than two arms' worth of runs on a clock | no clock at all, two time zones, or no split the runs admit |
+| `Vanished` | baseline under `VanishedMinBaselineSessions`, or no baseline slice | structurally none — an appearance is always readable |
+| `RetryMasked`, `RetryDeepening`, `RetryExhausted` | every run of the test was discounted as environmental | structurally none — an attempt number is always recorded |
+| `Flaky`, `AlwaysFailing`, `TimingOut` | every execution of the test was discounted | structurally none — an outcome is always readable |
+| `SharedFailure`, `BrokenFixture` | **absent** | **absent** |
+
+A published zero and an absence are different statements. A zero means the kind was offered tests and
+could read every one; an absence means the kind keeps no such tally.
+
+### Why five kinds publish nothing
+
+**`SharedFailure` and `BrokenFixture` are counted in signature groups, not tests.** Every other entry
+in the table is a count of fingerprints, and a count of groups published under the same field is not
+a number a reader can compare with the one beside it.
+
+**The three retry kinds share one figure rather than each publishing their own.** They read one
+reduction of a test's runs, and the chain that picks between them — exhausted, then deepening, then
+masked — stops at the first that fires. Whether a later kind could have been measured on a test an
+earlier one claimed is a question the algorithm never asks, and answering it for the tally alone would
+mean running all three on every fingerprint to fill in a number nobody reads. So the count is taken at
+the one precondition the three share: a test whose every run was an outage.
+
+### Two skips that are charged to nothing
+
+A fingerprint the index holds but cannot resolve to a test is an inconsistency inside the index, not a
+measurement the data declined; every provider passes over it silently.
+
+A test absent from the recent slice is passed over by `DurationProvider` for the reason the population
+matrix already gives — its absence belongs to `Vanished` — and counting it here would state one
+disappearance twice, in a line whose whole purpose is to name questions whose answers are missing.
+
+### Why it is per kind and never a total
+
+Summing across kinds counts one test once per question its data could not answer. Intersecting them
+collapses to nothing, because a test's pass and fail are always readable and so nearly every test is
+measured by something. And either total would move with `--kind`, which is the one thing a count of
+what could not be measured must not do: a reader comparing a full report with a narrowed one would
+read the difference as the suite improving. Per kind it cannot move, because the six providers own
+disjoint kind sets — a kind's figure comes from its own provider or the kind is absent from the map.
+
+### What the terms do not sum to
+
+For a kind that publishes both figures:
+
+```
+hypothesesTested[k] + notMeasured[k].AwaitingRuns + notMeasured[k].Unreadable
+    = the fingerprints that kind was offered
+```
+
+and the tests it measured and had nothing to say about are inside `hypothesesTested`, not published
+separately. That term is the common case — it is very nearly the suite — and publishing it would make
+the number the test count.
+
 ## Why the exceptions are exceptions
 
 **`SharedFailure` and `BrokenFixture` keep environmental sessions.** An environmental session *is* a

src/Xping.Cli/Report/Contract/EnvelopeBuilder.cs

@@ -7,6 +7,7 @@
 using System.Text.Json;
 using System.Text.Json.Nodes;
 using Xping.Cli.Report.Model;
+using Xping.Cli.Report.Providers;
 using Xping.Cli.Report.Windowing;
 
 namespace Xping.Cli.Report.Contract;
@@ -82,6 +83,7 @@ public static ReportEnvelope Build(
                 Math.Max(0, tests - flagged.Count),
                 result.ExcludedLowEvidence,
                 result.ExcludedNotSignificant,
+                NotMeasured(result),
 
                 context.EnvironmentalSessionCount,
                 context.PartialSessionCount,
@@ -92,6 +94,38 @@ [.. shown.Select(BuildFinding)],
             new TruncationDto(shown.Count, result.Findings.Count, DrillDown.ForFullReport()));
     }
 
+    /// <summary>
+    /// Projects the per-kind not-measured tally into the envelope's spelling.
+    /// </summary>
+    /// <param name="result">What the providers produced.</param>
+    /// <returns>One entry per kind that keeps such a tally, in <see cref="FindingKind"/> order.</returns>
+    /// <remarks>
+    /// Enumerated over the enum rather than over the dictionary, because a dictionary's order is
+    /// its insertion order and that is provider execution order — which the coordinator already
+    /// sorts, but only by provider name. Two runs over an unchanged store have to serialise
+    /// byte-identically, and taking the order from the declaration is what makes that a property of
+    /// the code rather than of a hash.
+    /// <para>
+    /// Keyed by <c>nameof</c> rather than by the enum. The registered converter camel-cases enum
+    /// values while <c>JsonSerializerOptions.DictionaryKeyPolicy</c> is unset, so a
+    /// <see cref="FindingKind"/> key would serialise as <c>durationRegression</c> and disagree with
+    /// the <c>DurationRegression</c> that every finding's <c>kind</c> and every <c>--kind</c>
+    /// argument spell.
+    /// </para>
+    /// </remarks>
+    private static Dictionary<string, NotMeasuredDto> NotMeasured(AnalysisResult result)
+    {
+        var published = new Dictionary<string, NotMeasuredDto>(StringComparer.Ordinal);
+
+        foreach (FindingKind kind in Enum.GetValues<FindingKind>())
+        {
+            if (result.NotMeasured.TryGetValue(kind, out NotMeasuredCount count))
+                published[kind.ToString()] = new NotMeasuredDto(count.AwaitingRuns, count.Unreadable);
+        }
+
+        return published;
+    }
+
     private static WindowDto BuildWindow(AnalysisWindow window) =>
         new(
             window.From,

src/Xping.Cli/Report/Contract/ReportEnvelope.cs

@@ -43,7 +43,7 @@ internal sealed record ReportEnvelope(
     /// <c>evidenceSessions</c> — the denominator <c>evidenceLevel</c> is banded from, which is the
     /// runs the finding was computed over rather than the runs its test appeared in.
     /// </remarks>
-    public const string CurrentSchemaVersion = "1.14";
+    public const string CurrentSchemaVersion = "1.15";
 }
 
 /// <summary>
@@ -84,13 +84,35 @@ internal sealed record ContextDto(string? Sha, string? Branch, string? Assembly)
 /// <param name="Tests">Distinct tests seen in the window.</param>
 /// <param name="Findings">Findings produced, before truncation.</param>
 /// <param name="Counts">Those findings broken down by severity.</param>
-/// <param name="Healthy">Tests no finding was raised about.</param>
+/// <param name="Healthy">
+/// Tests no finding was raised about. Not "tests that were checked and are fine": a test a metric
+/// could not be computed for is in here too, and <paramref name="NotMeasured"/> is what says so.
+/// The unmeasured are deliberately not subtracted — which of them there are depends on which kinds
+/// were asked for, and a headline count that moves with <c>--kind</c> would be worse than one that
+/// needs a second number read beside it.
+/// </param>
 /// <param name="ExcludedLowEvidence">Candidates dropped for resting on too little data.</param>
 /// <param name="ExcludedNotSignificant">
 /// Candidates dropped because their kind's comparison, charged for every fingerprint it ran on,
 /// no longer said anything. A large suite over a short window silences most of what it tests, and a
 /// reader given only an empty block cannot tell that from a suite with nothing to report.
 /// </param>
+/// <param name="NotMeasured">
+/// Per kind, the tests that kind could not be measured on at all, keyed by the kind as
+/// <c>findings[].kind</c> and <c>--kind</c> spell it.
+/// <para>
+/// Not a total and cannot be made into one. Adding the values counts a test once per question its
+/// data could not answer; a reader wanting "how much of my suite could this metric read" has to
+/// read one entry. The four kinds absent from the map keep no such tally — <c>SharedFailure</c> and
+/// <c>BrokenFixture</c> are counted in signature groups rather than tests, and the retry kinds are
+/// decided by a chain in which a later kind is never attempted once an earlier one fires. A present
+/// zero is a real statement: the kind was offered tests and read every one.
+/// </para>
+/// <para>
+/// Distinct from <paramref name="ExcludedLowEvidence"/>, which counts candidates a provider offered
+/// and the report then withheld. These are tests no candidate ever existed for.
+/// </para>
+/// </param>
 /// <param name="EnvironmentalSessions">Sessions discounted as environment failures.</param>
 /// <param name="PartialSessions">
 /// Sessions that covered only part of the suite — a <c>dotnet test --filter</c> run, or anything
@@ -108,12 +130,27 @@ internal sealed record SummaryDto(
     int Healthy,
     int ExcludedLowEvidence,
     int ExcludedNotSignificant,
+    IReadOnlyDictionary<string, NotMeasuredDto> NotMeasured,
     int EnvironmentalSessions,
     int PartialSessions,
     int IncompleteSessions,
     int UnreadableSessions,
     IReadOnlyList<string> FailedProviders);
 
+/// <summary>
+/// Why one kind could not be measured on some of the tests it was offered.
+/// </summary>
+/// <remarks>
+/// Two numbers because only one of them is answered by waiting. A test the comparison needs seven
+/// runs of and has four will have seven; a test whose every run recorded a zero median normalises
+/// nothing, and another such run normalises nothing either. A script deciding whether to keep
+/// collecting reads the first; a developer deciding whether their adapter is recording what the
+/// metric needs reads the second.
+/// </remarks>
+/// <param name="AwaitingRuns">Tests the kind needs more runs of.</param>
+/// <param name="Unreadable">Tests whose recorded data cannot answer this kind's question.</param>
+internal sealed record NotMeasuredDto(int AwaitingRuns, int Unreadable);
+
 /// <summary>
 /// One finding, with every value already at its published precision.
 /// </summary>

src/Xping.Cli/Report/FindingCoordinator.cs

@@ -3,6 +3,8 @@
  * License: [MIT]
  */
 
+using System.Collections.ObjectModel;
+
 using Xping.Cli.Report.Model;
 using Xping.Cli.Report.Providers;
 using Xping.Cli.Report.Scoring;
@@ -21,14 +23,34 @@ namespace Xping.Cli.Report;
 /// eleven candidates and one that had nothing to discard are the same empty block, and a reader who
 /// cannot tell them apart learns the wrong thing from silence.
 /// </param>
+/// <param name="NotMeasured">
+/// Per kind, the fingerprints that kind could not be measured on at all, split by whether waiting
+/// fixes it.
+/// <para>
+/// A different quantity from the two counts above, and deliberately not added to them. Those count
+/// candidates a provider offered and this pass then dropped; this counts fingerprints no candidate
+/// was ever offered for, because the provider could compute nothing about them. Only the first kind
+/// of drop is a judgement the report made.
+/// </para>
+/// <para>
+/// <b>Per kind, and never totalled.</b> Summing across kinds would count one test as many times as
+/// there are questions its data could not answer; intersecting them collapses to nothing, because
+/// a test's pass and fail are always readable and so nearly every test is measured by something.
+/// Worse, either total would move with <c>--kind</c>, which is the one thing a count of what could
+/// not be measured must not do. Per kind it cannot: the six providers own disjoint kind sets, so a
+/// kind's figure comes from its own provider or the kind is absent from the map.
+/// </para>
+/// </param>
 internal sealed record AnalysisResult(
     IReadOnlyList<Finding> Findings,
     IReadOnlyList<string> FailedProviders,
     int ExcludedLowEvidence,
-    int ExcludedNotSignificant)
+    int ExcludedNotSignificant,
+    IReadOnlyDictionary<FindingKind, NotMeasuredCount> NotMeasured)
 {
     /// <summary>Gets an empty result.</summary>
-    public static AnalysisResult Empty { get; } = new([], [], 0, 0);
+    public static AnalysisResult Empty { get; } =
+        new([], [], 0, 0, ReadOnlyDictionary<FindingKind, NotMeasuredCount>.Empty);
 }
 
 /// <summary>
@@ -47,6 +69,14 @@ internal sealed record AnalysisResult(
 /// more to the point — can it judge its own p-values against the number of times it produced one.
 /// It reports the size of the family it tested and this decides what that family may claim.
 /// </para>
+/// <para>
+/// What a provider may report about itself, on the other side of that line, is what it observed and
+/// what it could not — never what either is worth. <see cref="AnalysisResult.NotMeasured"/> is a
+/// description of coverage and not a gate on emission: no candidate appears or disappears because
+/// of it, exactly as no finding's severity moves because of the denominator a provider publishes
+/// beside it. The rule the division protects is that a test flagged by one metric is never silently
+/// dropped by another, and a count of unanswered questions cannot drop anything.
+/// </para>
 /// </remarks>
 internal sealed class FindingCoordinator(IEnumerable<IFindingProvider> providers)
 {
@@ -68,6 +98,7 @@ public AnalysisResult Run(
         // provider has run.
         var surviving = new List<FindingCandidate>();
         var tested = new Dictionary<FindingKind, int>();
+        var notMeasured = new Dictionary<FindingKind, NotMeasuredCount>();
 
         int lowEvidence = 0;
 
@@ -102,6 +133,20 @@ public AnalysisResult Run(
                     tested[kind] = tested.GetValueOrDefault(kind) + count;
             }
 
+            // The same filter, deliberately. `--kind` has to narrow a kind's family, its candidates
+            // and the tally of what it could not measure together, or the report answers "how much
+            // of the suite could this metric read" differently according to what else was asked for
+            // in the same invocation.
+            //
+            // The addition never actually adds across providers, because each kind is owned by
+            // exactly one of them. It is written as an addition anyway so that a provider split in
+            // two later cannot silently overwrite half its own answer.
+            foreach ((FindingKind kind, NotMeasuredCount count) in report.NotMeasured)
+            {
+                if (kinds == null || kinds.Contains(kind))
+                    notMeasured[kind] = notMeasured.GetValueOrDefault(kind) + count;
+            }
+
             foreach (FindingCandidate candidate in report.Candidates)
             {
                 if (kinds != null && !kinds.Contains(candidate.Kind))
@@ -162,7 +207,7 @@ public AnalysisResult Run(
         findings.Sort(FindingOrder.Instance);
         failed.Sort(StringComparer.Ordinal);
 
-        return new AnalysisResult(findings, failed, lowEvidence, notSignificant);
+        return new AnalysisResult(findings, failed, lowEvidence, notSignificant, notMeasured);
     }
 
     /// <summary>