fix(cli): ask only the runs that covered the suite whether a test stopped

docs/cli/command-reference.md

@@ -238,6 +238,17 @@ it eases towards 0.632 as the history lengthens. On a window shorter than eight
 slice narrows to a single run and one run's absence never reaches the bar, so the kind is silent
 there — see [known limitations](../known-limitations.md).
 
+It also asks only the runs that were in a position to answer. A run under a `dotnet test --filter`
+did not fail to see the tests it excluded; it never looked for them, and counting its silence makes
+every unselected test look deleted. A run whose distinct tests are under half those of the largest
+run in the window is therefore set aside, the rest are re-split into their own earlier and current
+runs, and the finding's denominators count only those — the sentence says `full runs` and a
+`set aside` metric gives the number left out. The summary line reports how many runs covered part of
+the suite. Only this kind sets them aside; every other kind still counts them in full, because a
+filtered run's outcomes are as true as any other run's. The cost is that a deletion removing more
+than half a suite is indistinguishable from a filter and is not reported — see
+[known limitations](../known-limitations.md).
+
 ### Finding ids
 
 The `f_…` on each finding is a short, stable identity for that finding — a hash of what the
@@ -328,7 +339,7 @@ Every finding carries a `headline` — the same sentence the rendered report pri
 
 ```json
 {
-  "schemaVersion": "1.12",
+  "schemaVersion": "1.13",
   "window": { "sessionCount": 20, "resolution": "default", "currentSliceSize": 3 },
   "context": { "sha": "a3f9c2e", "branch": "main", "assembly": "Checkout.Tests" },
   "summary": {
@@ -337,7 +348,8 @@ Every finding carries a `headline` — the same sentence the rendered report pri
     "counts": { "high": 1, "medium": 2, "low": 0 },
     "healthy": 409,
     "excludedLowEvidence": 41,
-    "excludedNotSignificant": 6
+    "excludedNotSignificant": 6,
+    "partialSessions": 0
   },
   "findings": [
     {

docs/known-limitations.md

@@ -319,6 +319,44 @@ anything stopped — but it does mean a fresh store says nothing about deleted t
 eight runs in it. The run rate and the p-value are published with every finding that does clear the
 bar.
 
+---
+
+### `Vanished` Cannot Tell A Filtered Run From A Deletion, So It Trusts Neither Below Half A Suite
+
+**Impact**: a run that covered less than half the suite is counted on neither side of an absence, so
+a deletion that removed more than half a suite is never reported. A `--filter` selecting *more* than
+half the suite still produces one false `stopped running` per unselected test.
+
+**Reason**: a run under a `dotnet test --filter` did not fail to see the tests it excluded — it never
+looked for them. Counting its silence makes every unselected test look deleted, which is what an
+ordinary inner loop produces in about ten minutes: a handful of full runs, then a stream of filtered
+ones, then a report claiming most of the suite has stopped running. Every statement in it is true of
+the data and false about the world.
+
+So the report classifies each run by how much of the suite it covered — its distinct tests against
+the largest run in the window, which is the best evidence the window holds of how big the suite is,
+and the only anchor a store of mostly filtered runs does not corrupt. The median does not work: four
+full runs and sixteen filtered ones has a median of one test, and every run in it measures as
+typical. A report is scoped to exactly one assembly, so the runs being compared are always runs of
+the same suite. Runs covering less than half are set aside, the remaining runs are re-split into
+their own "now" and "before", and the counts on the finding are of those runs alone — `full runs` in
+the sentence, with a `set aside` metric saying how many were left out.
+
+**The trade, stated rather than discovered**: a count cannot separate the two cases, and no threshold
+makes it able to. Nine tests missing from a suite of seventeen is the same table whether they were
+excluded or removed. The line is placed at a half — the point at which a run stopped being a run of
+the suite and became a run of part of it — and it is deliberately biased towards silence: `Vanished`
+is capped at `Severity.Low` because a disappearance is usually something the developer did on purpose
+a minute ago, so a missed one costs little, whereas the false positive arrives once per unselected
+test on every filtered run for as long as it stays in the window.
+
+**Related**: setting runs aside shortens the history this kind measures against, so a store whose
+runs are mostly filtered can fall below the eight runs the section above requires and report nothing
+at all. Only the kinds that read absence set these runs aside — every other kind still counts them in
+full, because a filtered run's *outcomes* are as true as any other run's and it is only its silences
+that mean nothing. The summary line says how many runs covered part of the suite, so the distinction
+is visible rather than inferred.
+
 ### `RetryExhausted` Is Observed, And The Declared Retry Limit Is Not Interpreted
 
 **Impact**: a test whose retry attribute allows three retries but which only ever recorded two
@@ -491,3 +529,4 @@ When reporting, please include:
 | 1.7.0   | Documented what `TimeSensitive` now charges for searching three axes, and what that costs |
 | 1.8.0   | Documented what `ParallelSensitive` now measures, and the duration confound it cannot correct |
 | 1.9.0   | Documented the run rate `Vanished` now requires, and the window size below which it is silent |
+| 1.10.0  | Documented how `Vanished` treats a run that covered part of the suite, and what that trade costs |

src/Xping.Cli/Report/AnalysisContext.cs

@@ -44,9 +44,28 @@ public AnalysisContext(AnalysisWindow window, RevisionContext? revision)
         for (int position = 0; position < window.Sessions.Count; position++)
             views.Add(SessionView.For(window.Sessions[position], position));
 
+        // Partiality is a comparison and not a measurement, so it needs the whole window before any
+        // one session can be classified. The anchor is the largest run in it: the best evidence the
+        // window holds of how big the suite is, and the only one a store of mostly filtered runs
+        // does not corrupt — see LocalAnalysisConstants.PartialSessionShare.
+        int suiteSize = 0;
+        foreach (SessionView view in views)
+            suiteSize = Math.Max(suiteSize, view.Tests);
+
+        if (suiteSize > 0)
+        {
+            double floor = LocalAnalysisConstants.PartialSessionShare * suiteSize;
+            for (int position = 0; position < views.Count; position++)
+            {
+                if (views[position].Tests < floor)
+                    views[position] = views[position] with { IsPartial = true };
+            }
+        }
+
         SessionViews = views;
         _viewsBySession = views.ToDictionary(v => v.Session.SessionId);
         EnvironmentalSessionCount = views.Count(v => v.IsLikelyEnvironmental);
+        PartialSessionCount = views.Count(v => v.IsPartial);
     }
 
     /// <summary>Gets the sessions under analysis and the boundaries that produced them.</summary>
@@ -71,6 +90,17 @@ public AnalysisContext(AnalysisWindow window, RevisionContext? revision)
     /// </remarks>
     public int EnvironmentalSessionCount { get; }
 
+    /// <summary>
+    /// Gets how many analysed sessions covered only part of the suite.
+    /// </summary>
+    /// <remarks>
+    /// Reported in the summary as an observation rather than a discount. Only the kinds that read
+    /// absence set these sessions aside — every other kind reads outcomes of executions that
+    /// happened, and a filtered run's outcomes are as true as any other's — so the line says what
+    /// the window contains and does not claim the numbers were adjusted.
+    /// </remarks>
+    public int PartialSessionCount { get; }
+
     /// <summary>
     /// Gets the health of one analysed session.
     /// </summary>

src/Xping.Cli/Report/Contract/EnvelopeBuilder.cs

@@ -84,6 +84,7 @@ public static ReportEnvelope Build(
                 result.ExcludedNotSignificant,
 
                 context.EnvironmentalSessionCount,
+                context.PartialSessionCount,
                 incompleteSessions,
                 unreadableSessions,
                 result.FailedProviders),

src/Xping.Cli/Report/Contract/EvidenceHeadline.cs

@@ -536,27 +536,57 @@ private static (string, IReadOnlyList<MetricDto>) TimeSensitive(TimeSensitiveEvi
     /// <param name="e">The evidence.</param>
     /// <returns>The headline and its metrics.</returns>
     /// <remarks>
+    /// <para>
     /// The headline carries the denominator and not the p-value, because here the denominator is the
     /// discriminating figure: "3 of 17" and "17 of 17" are visibly different claims to a reader
     /// skimming the fence, where the two arms of a split are not. The p-value is a metric, for the
     /// reader who opens the finding to check how much belief the sentence earned.
+    /// </para>
+    /// <para>
+    /// Both denominators count only the runs that covered the suite, so where any were set aside
+    /// both clauses say "full runs" and a metric says how many were left out. Both, because "the
+    /// last 3" and "the last 3 full runs" are not the same three runs once anything has been set
+    /// aside, and a clause that is only true given the one before it is the kind a reader pastes
+    /// into a chat window on its own. Qualified rather than worded that way always, because in a
+    /// store with no filtered runs there is nothing for the word to distinguish the runs from and
+    /// the shorter sentence is the true one. The wording cannot destabilise the finding's id, which
+    /// hashes the kind and the subject and nothing else — see <see cref="Model.FindingId"/>.
+    /// </para>
     /// </remarks>
-    private static (string, IReadOnlyList<MetricDto>) Vanished(VanishedEvidence e) =>
-    (
-        $"ran in {e.BaselineSessions} of {e.BaselineSessionCount} earlier runs, " +
-        $"absent from the last {e.CurrentSessionCount}",
+    private static (string, IReadOnlyList<MetricDto>) Vanished(VanishedEvidence e)
+    {
+        bool anySetAside = e.PartialSessionsSetAside > 0;
+        string runs = anySetAside ? "full runs" : "runs";
+
+        List<MetricDto> metrics =
         [
             new(
                 "ran in",
-                $"{e.BaselineSessions} of {e.BaselineSessionCount} earlier runs " +
+                $"{e.BaselineSessions} of {e.BaselineSessionCount} earlier {runs} " +
                 $"({Percent(e.BaselineRunRate)})"),
-            new("absent from", $"the last {e.CurrentSessionCount} runs"),
+            new("absent from", $"the last {e.CurrentSessionCount} {runs}"),
             new("executions", e.ExecutionsInWindow.ToString(CultureInfo.InvariantCulture)),
 
             // One-sided, and legitimately so: the kind only ever forms a table for a test already
             // absent, so the direction was fixed before the counts were.
             new("significance", $"p {Probability(e.PValue)} one-sided")
-        ]);
+        ];
+
+        // Only where there were any. A "0 set aside" line on every finding in every ordinary store
+        // would be noise standing in for the absence of a caveat.
+        if (anySetAside)
+            metrics.Add(new("set aside", $"{Runs(e.PartialSessionsSetAside)} that covered part of the suite"));
+
+        // The trailing noun only where it discriminates. On an ordinary store "the last 3" can mean
+        // nothing but the last three runs, and the sentence is the one this kind has always printed.
+        string absence = anySetAside
+            ? $"absent from the last {e.CurrentSessionCount} {runs}"
+            : $"absent from the last {e.CurrentSessionCount}";
+
+        return (
+            $"ran in {e.BaselineSessions} of {e.BaselineSessionCount} earlier {runs}, {absence}",
+            metrics);
+    }
 
     private static string Times(int count) =>
         count == 1 ? "once" : $"{count.ToString(CultureInfo.InvariantCulture)} times";

src/Xping.Cli/Report/Contract/ReportEnvelope.cs

@@ -39,14 +39,12 @@ internal sealed record ReportEnvelope(
     /// Moves whenever anything a consumer reads changes shape, and the per-kind evidence payloads
     /// are part of that even though this document describes them as opaque: a script that reached
     /// into <c>evidence</c> for a field this build no longer emits is reading a contract, and
-    /// leaving the number still would tell it nothing had moved. 1.12 is where every evidence
-    /// record renamed the fields a rate is taken over — <c>executionsConsidered</c> and
-    /// <c>sessionsConsidered</c> where discounting applied, <c>executionsInWindow</c> and
-    /// <c>sessions</c> where it did not — split <c>discountedExecutions</c> into its environmental
-    /// and clustered halves, and where the finding gained <c>population</c>, which says which of
-    /// those the numbers beside it describe.
+    /// leaving the number still would tell it nothing had moved. 1.13 is where the summary gained
+    /// <c>partialSessions</c> and <c>VanishedEvidence</c> gained <c>partialSessionsSetAside</c> —
+    /// the runs that covered only part of the suite, which an absence is counted on neither side
+    /// of.
     /// </remarks>
-    public const string CurrentSchemaVersion = "1.12";
+    public const string CurrentSchemaVersion = "1.13";
 }
 
 /// <summary>
@@ -95,6 +93,12 @@ internal sealed record ContextDto(string? Sha, string? Branch, string? Assembly)
 /// reader given only an empty block cannot tell that from a suite with nothing to report.
 /// </param>
 /// <param name="EnvironmentalSessions">Sessions discounted as environment failures.</param>
+/// <param name="PartialSessions">
+/// Sessions that covered only part of the suite — a <c>dotnet test --filter</c> run, or anything
+/// else that ran a fraction of the tests the window's largest run did. An observation and not a
+/// discount: only the kinds that read absence set such a session aside, because a filtered run's
+/// outcomes are as true as any other run's and it is only its silences that mean nothing.
+/// </param>
 /// <param name="IncompleteSessions">Sessions found but not finalised.</param>
 /// <param name="UnreadableSessions">Session files that could not be read.</param>
 /// <param name="FailedProviders">Metrics that threw and produced nothing.</param>
@@ -106,6 +110,7 @@ internal sealed record SummaryDto(
     int ExcludedLowEvidence,
     int ExcludedNotSignificant,
     int EnvironmentalSessions,
+    int PartialSessions,
     int IncompleteSessions,
     int UnreadableSessions,
     IReadOnlyList<string> FailedProviders);

src/Xping.Cli/Report/Indexes/SessionView.cs

@@ -21,6 +21,11 @@ namespace Xping.Cli.Report.Indexes;
 /// It is a classification against two thresholds, not a claim about what went wrong. The report says
 /// a session looks environmental; it never says why, and it never says so about a test.
 /// </para>
+/// <para>
+/// <see cref="IsPartial"/> is the other flag, and it answers a different question: not whether the
+/// session's outcomes can be trusted, but whether its silences can. It is window-relative, so
+/// <see cref="For"/> cannot decide it — see <see cref="AnalysisContext.PartialSessionCount"/>.
+/// </para>
 /// </remarks>
 /// <param name="Session">The session itself.</param>
 /// <param name="Index">Its position in the window; 0 is the newest.</param>
@@ -38,6 +43,30 @@ internal sealed record SessionView(
     double FailureRate,
     bool IsLikelyEnvironmental)
 {
+    /// <summary>
+    /// Gets a value indicating whether the session covered only part of the suite.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// A run under a <c>dotnet test --filter</c> is not a run in which the tests it excluded failed
+    /// to appear — it is a run that never asked about them. A kind reading absence has to set such a
+    /// session aside, or every unselected test looks deleted.
+    /// </para>
+    /// <para>
+    /// Not a positional member, because it cannot be measured from one session: it is
+    /// <see cref="Tests"/> against <see cref="LocalAnalysisConstants.PartialSessionShare"/> of the
+    /// largest run in the window, and the window is what <see cref="For"/> does not have.
+    /// <see cref="AnalysisContext"/> sets it once the whole window is measured, which is also the
+    /// only place it means anything.
+    /// </para>
+    /// <para>
+    /// A classification against a threshold and not a claim about what happened, in the same way
+    /// <see cref="IsLikelyEnvironmental"/> is. The report says a session covered part of the suite;
+    /// it never says a filter was the reason, because a deletion produces the same count.
+    /// </para>
+    /// </remarks>
+    public bool IsPartial { get; init; }
+
     /// <summary>
     /// Measures one session.
     /// </summary>

src/Xping.Cli/Report/LocalAnalysisConstants.cs

@@ -77,6 +77,43 @@ internal static class LocalAnalysisConstants
     /// </summary>
     public const int SmallWindowSessionCount = 8;
 
+    /// <summary>
+    /// Share of the window's largest run a session must cover to count as a run of the suite (0.50).
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// A <c>dotnet test --filter</c> run is an ordinary thing to find in a local store, and a
+    /// session that ran one test of seventeen says nothing whatever about the other sixteen. Absence
+    /// is only evidence where the run was in a position to produce a presence, so a session covering
+    /// a small part of the suite is set aside by the kinds that read absence rather than outcome —
+    /// today that is <see cref="Providers.VanishedProvider"/> alone.
+    /// </para>
+    /// <para>
+    /// The anchor is the largest session in the window and deliberately not its median. A store of
+    /// four full runs and sixteen filtered ones — which is what ten minutes of an inner loop
+    /// produces — has a median of one test, and every session in it would measure as typical
+    /// against that. The largest run is the best evidence the window holds of how big the suite
+    /// actually is, and a report is scoped to exactly one assembly, so the sessions being compared
+    /// are always runs of the same suite.
+    /// </para>
+    /// <para>
+    /// A count cannot tell a filtered run from a deletion, and no threshold makes it able to: a run
+    /// of eight tests where the suite has seventeen is the same table whether nine tests were
+    /// excluded or removed. So the line is a trade rather than a discovery, and it is placed at a
+    /// half — the point at which a run stopped being a run of the suite and became a run of part of
+    /// it. What that costs, in both directions: a filter selecting more than half the suite still
+    /// produces false absences, and a deletion of more than half a suite is never reported at all.
+    /// </para>
+    /// <para>
+    /// Biased towards silence on purpose. <see cref="Model.FindingKind.Vanished"/> is capped at
+    /// <see cref="Model.Severity.Low"/> because a disappearance is usually something the developer
+    /// did on purpose thirty seconds ago, so a missed one costs almost nothing; whereas the false
+    /// positive arrives one per unselected test, on every filtered run, for as long as the run stays
+    /// in the window.
+    /// </para>
+    /// </remarks>
+    public const double PartialSessionShare = 0.50;
+
     /// <summary>
     /// Failure rate at or above which a test is broken rather than flaky (0.90).
     /// </summary>