Carry gaps never searched into the evolutionary audit and run comparison

CHANGELOG.md

@@ -9,6 +9,16 @@ and Recore uses semantic versioning.
 
 ### Fixed
 
+- `evolutionary_events.tsv` no longer presents a gain or loss built on gaps
+  Recore never searched as an audited call. New columns `absent_not_searched`
+  and `not_searched_reason` count those absences and say why; a call none of
+  whose absences were searched drops one support level (and so in score and
+  priority), and every affected call is tagged in `mechanism_detail` and
+  explained in `recommended_wording`.
+- `benchmarks/compare_runs.py` counted `NOT_SEARCHED_*` cells as calls, so a
+  comparison against a run from before those states listed every unsearched
+  cell as a new correction. They are absences now, and a cell searched in one
+  run and not the other is `search_coverage_changed`.
 - A gap Recore never searched no longer reads like one it searched and found
   empty. The states file writes `NOT_SEARCHED_LOW_PREVALENCE`,
   `NOT_SEARCHED_NO_CLUSTER_SEQUENCE` or `NOT_SEARCHED_NO_FASTA` where it wrote

benchmarks/README.md

@@ -112,7 +112,10 @@ python3 benchmarks/compare_runs.py --old old_recore_output --new new_recore_outp
 
 The comparison uses the union of matrix cells, so clusters or genomes present
 in only one run are reported as `old_only_cell` / `new_only_cell` instead of
-being silently ignored.
+being silently ignored. A cell absent in both runs but searched in only one of
+them (an `ABSENT` against a `NOT_SEARCHED_*`, as older runs wrote `ABSENT` for
+both) is `search_coverage_changed`, not a correction, and is counted apart from
+`changed_cells`.
 
 Generate a compact Markdown/HTML report:
 

benchmarks/compare_runs.py

@@ -72,19 +72,32 @@ def read_evolution(run_dir: Path) -> dict[str, dict[str, str]]:
         return {row["gene_cluster"]: row for row in reader}
 
 
+def is_absent(state: str) -> bool:
+    """No presence and no evidence: a gap searched and found empty, or one that
+    was never searched. A NOT_SEARCHED_* state is an absence, and counting it as
+    a call made every unsearched cell a "new correction" against older runs,
+    which wrote ABSENT for both."""
+    return state in {"", "ABSENT"} or state.startswith("NOT_SEARCHED")
+
+
 def classify_change(old_state: str, new_state: str) -> str:
     if old_state == "MISSING_CELL":
         return "new_only_cell"
     if new_state == "MISSING_CELL":
         return "old_only_cell"
-    old_present = old_state not in {"", "ABSENT"}
-    new_present = new_state not in {"", "ABSENT"}
     if old_state == new_state:
         return "same"
+    old_present = not is_absent(old_state)
+    new_present = not is_absent(new_state)
     if not old_present and new_present:
         return "new_correction"
     if old_present and not new_present:
         return "lost_correction"
+    if not old_present and not new_present:
+        # Absent in both runs, but searched in one and not the other, or left
+        # out of the search for a different reason: the search changed, the
+        # matrix did not.
+        return "search_coverage_changed"
     return "state_changed"
 
 
@@ -128,6 +141,7 @@ def main() -> int:
             "old_only_cell",
             "new_only_cell",
             "changed_cells",
+            "search_coverage_changed",
         ])
         for cluster in sorted(cluster_counts):
             counts = cluster_counts[cluster]
@@ -138,7 +152,7 @@ def main() -> int:
                 + counts["old_only_cell"]
                 + counts["new_only_cell"]
             )
-            if changed:
+            if changed or counts["search_coverage_changed"]:
                 writer.writerow([
                     cluster,
                     counts["same"],
@@ -148,6 +162,7 @@ def main() -> int:
                     counts["old_only_cell"],
                     counts["new_only_cell"],
                     changed,
+                    counts["search_coverage_changed"],
                 ])
 
     old_evo = read_evolution(args.old)
@@ -199,6 +214,7 @@ def main() -> int:
             "state_changed",
             "old_only_cell",
             "new_only_cell",
+            "search_coverage_changed",
         ]:
             writer.writerow([key, cell_counts[key]])
 

docs/guide/outputs.md

@@ -213,6 +213,7 @@ corrected binary state for each cluster.
 |--------|-------------|
 | `gene_cluster` | Panaroo cluster name |
 | `n_present_corrected`, `prevalence_corrected` | Presence after accepted matrix corrections and NSC presence updates |
+| `absent_not_searched`, `not_searched_reason` | How many of the cluster's absences in the corrected matrix were never searched, and why: `low_prevalence` (the cluster is below `--min-prevalence`), `no_cluster_sequence`, or `no_fasta` (some genomes have no FASTA); `0` and blank when every absence was searched |
 | `tree_matched_tips` | Number of tree tips matched to matrix genomes; blank when no tree is provided |
 | `reference_state` | `present`, `absent`, or `not_assessed` for the requested `--reference-genome` |
 | `phylogenetic_pattern` | `ancestral_retained`, `ancestral_loss`, `lineage_specific_gain`, `clade_associated_presence`, `clade_associated_presence_with_losses`, `recurrent_gain`, `recurrent_loss`, `homoplastic_gain_loss`, `ambiguous_gain_loss`, `singleton_or_noise`, `dispersed_gain_loss`, or `not_assessed_no_tree` |
@@ -224,7 +225,7 @@ corrected binary state for each cluster.
 | `carrier_monophyly`, `max_carrier_clade_fraction` | Whether all carrier tips form one pure carrier-only clade, and the largest carrier-only clade fraction |
 | `carrier_mrca_tips`, `carrier_mrca_purity` | Size and carrier purity of the smallest tree clade containing all carrier tips; low purity means a single-gain story would require many internal losses |
 | `molecular_mechanism` | Best conservative mechanism call: reference-derived allele, IS interruption, pseudogenization, fragmentation, duplication/paralog expansion, context shift, acquisition/insertion candidate, deletion/loss candidate, homoplastic/unstable signal, ancestral retention, or unresolved |
-| `mechanism_detail`, `support_level`, `evolutionary_signal_score`, `highlight_priority`, `recommended_wording` | Evidence summary, integrated prioritization, and publication-facing interpretation. With `--nscluster`, `mechanism_detail` also carries `nsc_representative` on the cluster that receives the merged support and `nsc_member_merged_into_representative` on each redundant member |
+| `mechanism_detail`, `support_level`, `evolutionary_signal_score`, `highlight_priority`, `recommended_wording` | Evidence summary, integrated prioritization, and publication-facing interpretation. With `--nscluster`, `mechanism_detail` also carries `nsc_representative` on the cluster that receives the merged support and `nsc_member_merged_into_representative` on each redundant member. A gain or loss call that rests on absences never searched carries `absences_not_searched` (none searched, and one support level lower) or `absences_not_searched=N/M` (some), and its wording says so; see [Evolutionary events](../how-it-works/evolutionary-events.md#calls-that-rest-on-gaps-recore-never-searched) |
 | `cluster_component_id`, `cluster_component_size`, `cluster_component_role`, `cluster_component_relations`, `cluster_component_recommendation` | Connected-component context for split, annotation-matched, NSC, and paralog-linked clusters |
 | `is_interruption_supported`, `mobile_element_*` | Nearby mobile element signal. `mobile_element_source=sequence` comes from `--mobile-element-reference` alignments and is name-agnostic; `mobile_element_source=annotation` is the fallback from GFF labels/products/notes. IS calls are conservative and require sequence support or an insertion-sequence-like label within 250 bp of a fragmented/disrupted rescue |
 | `reference_event_class`, `target_reference_cluster`, `target_reference_gene` | Reference-derived event evidence when available |

docs/how-it-works/evolutionary-events.md

@@ -119,7 +119,9 @@ For manuscript highlighting, the safest candidates are usually clusters with:
 - coherent `phylogenetic_pattern` with low/moderate blink flag;
 - high `carrier_mrca_purity` or `dollo_origin_purity`;
 - no paralog/component warning;
-- compatible synteny and clear ORF integrity.
+- compatible synteny and clear ORF integrity;
+- `absent_not_searched` of 0, so every absence the call rests on was searched
+  (see below).
 
 Clusters with `duplication_or_paralog_expansion`,
 `annotation_or_clustering_artifact`,
@@ -131,6 +133,34 @@ the locus is inspected manually. Clusters with `reference_gene_*` or
 reference-absent genes, even if the original pangenome matrix marked them as
 absent from the reference.
 
+## Calls that rest on gaps Recore never searched
+
+Gains and losses are made of absences: a gain needs the gene missing in the
+rest of the tree, a loss needs it missing where it was lost. A gap outside the
+search (a cluster below `--min-prevalence`, a cluster with no sequence to search
+with, or a genome with no FASTA) keeps the input matrix's call, so a call built
+on such absences is only as good as that call. Under the default prevalence
+floor that is every gain on a cluster in fewer than one genome in ten.
+
+`absent_not_searched` counts those absences and `not_searched_reason` says why
+(`low_prevalence`, `no_cluster_sequence` or `no_fasta`). For the calls that rest
+on absences (`clean_insertion_or_acquisition_candidate`,
+`reference_or_lineage_loss_candidate`, `deletion_loss_candidate`,
+`unstable_or_homoplastic_accessory_signal` and
+`unresolved_accessory_variation`):
+
+- when none of the absences behind the call were searched (for a reference
+  loss, when the reference genome's was not), `support_level` drops one level,
+  `mechanism_detail` gains `absences_not_searched`, and `recommended_wording`
+  says so. The lower support lowers `evolutionary_signal_score`, and with it
+  `highlight_priority`;
+- when only some were not searched, `mechanism_detail` gains
+  `absences_not_searched=N/M` and the wording says how many, but the searched
+  absences still carry the call.
+
+Calls made from rescue evidence (pseudogenes, fragments, paralogs, reference
+events) do not rest on absences and are left as they are.
+
 ## Where the fields live
 
 Every field named on this page is a column of `evolutionary_events.tsv`; its

src/pipeline.rs

@@ -1058,6 +1058,7 @@ pub fn run(args: &Args) -> Result<()> {
         reference_genome: args.reference_genome.as_deref(),
         nsc_result: &nsc_result,
         component_info: &component_info,
+        coverage: &coverage,
         output_dir: &args.output,
     })?;
     if args.write_corrected_gff {