Score sequence-free libraries with library-wide chemistry fallbacks

docs/development.md

@@ -22,11 +22,29 @@ and target JSON. Format detection also inspects contents; the
 
 `timsseek` and `timsquery_viewer` use the same registry through `ReferenceLibrary`.
 Every loaded scoring library contains geometry usable for query extraction.
-The current scoring bridge requires ion-annotated fragments and reference
-fragment intensities; extraction also accepts opaque fragment labels. The
-query reader's target-list JSON schemas (`Target` and `ElutionGroupInput` arrays)
-currently load geometry without a reference-intensity sidecar. This describes
-those reader paths, not a restriction of JSON as an encoding.
+Scoring requires retained fragments and aligned reference intensities. mzSpecLib
+can supply these without sequence or fragment annotations. Opaque peaks disable
+fragment-isotope scores and automatic mass-shift decoy generation library-wide;
+supplied decoys remain usable. Without decoys, search scores the full acquisition
+RT range and writes raw scores, omitting competition, discriminant-score and
+q-value columns (`result_mode=raw` Parquet metadata). It bypasses calibration,
+rescoring and q-value filtering. Supplied decoys do not by themselves validate a
+decoy strategy for a new analyte class.
+
+`run_report.json` records the resolved `scoring_plan` once for the search library,
+plus `calibration_scoring_plan` when a separate calibration library is supplied.
+These are the same plans serialized in Parquet metadata: sequence-operation
+coverage, precursor-isotope method/reasons, fragment-isotope availability, and
+`decoys` (requested policy, resolved strategy, reason, stored-decoy count).
+Per-file `pipeline.raw_scores` records whether raw scoring was used.
+
+
+Programmatic `TargetTable::Str` accepts an optional intensity sidecar. Scoring
+assigns unique packed unknown keys (currently at most 255 peaks per entry),
+preserving the original opaque labels separately; even a string such as `y3`
+is not interpreted as chemistry. The query reader's target-list JSON schemas
+(`Target` and `ElutionGroupInput` arrays) still supply geometry without reference
+intensities, so those reader routes remain extraction-only.
 
 Standalone `calib_dash` reads saved `calibration.json`, not a spectral library.
 

rust/timsquery/src/models/capabilities.rs

@@ -10,10 +10,10 @@ pub struct TargetCapabilities {
     pub decoys: DecoyStrategy,
 }
 
-/// Runtime reflection of whether this arena's label carries ion chemistry
-/// (`FragmentLabel`). `IonAnnot` arenas => `Available`; string-labelled
-/// arenas => `Unavailable`. Sequence-operation eligibility is resolved from
-/// stored analyte facts by the scoring library.
+/// Whether the label representation can carry ion chemistry. `IonAnnot` may
+/// still contain unknown placeholders; scoring inspects actual labels across
+/// the whole library before enabling operations. Sequence-operation eligibility
+/// is independently resolved from stored analyte facts.
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
 pub enum FragmentFeatureState {
     Available,
@@ -34,7 +34,8 @@ pub enum IsotopeStrategy {
 /// and generates none is [`Stored`](Self::Stored) with
 /// `n_stored_decoys() == 0`; that is a property of the rows, not of the
 /// strategy, and reporting it belongs to whoever counts rows.
-#[derive(Debug, Clone, Copy, PartialEq)]
+#[derive(Debug, Clone, Copy, PartialEq, Serialize)]
+#[serde(tag = "method", rename_all = "snake_case")]
 pub enum DecoyStrategy {
     /// Score the stored rows as they are. Any decoys the file shipped are the
     /// decoys; nothing is derived.
@@ -45,6 +46,51 @@ pub enum DecoyStrategy {
     MassShift { offset: f64 },
 }
 
+/// Why sealing selected its decoy strategy. Preserved for downstream reports.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
+#[serde(rename_all = "snake_case")]
+pub enum DecoyResolutionReason {
+    PolicyNever,
+    SuppliedDecoys,
+    MissingFragmentChemistry,
+    FragmentChemistryAvailable,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Serialize)]
+pub struct DecoyResolution {
+    pub requested: DecoyPolicy,
+    pub strategy: DecoyStrategy,
+    pub reason: DecoyResolutionReason,
+    pub stored_decoys: usize,
+}
+
+impl DecoyResolution {
+    pub(crate) fn resolve(
+        requested: DecoyPolicy,
+        stored_decoys: usize,
+        generation_supported: bool,
+    ) -> Self {
+        let mut strategy = requested.strategy(stored_decoys > 0);
+        let reason = match (requested, strategy) {
+            (DecoyPolicy::Never, _) => DecoyResolutionReason::PolicyNever,
+            (_, DecoyStrategy::Stored) => DecoyResolutionReason::SuppliedDecoys,
+            (_, DecoyStrategy::MassShift { .. }) if generation_supported => {
+                DecoyResolutionReason::FragmentChemistryAvailable
+            }
+            _ => {
+                strategy = DecoyStrategy::Stored;
+                DecoyResolutionReason::MissingFragmentChemistry
+            }
+        };
+        Self {
+            requested,
+            strategy,
+            reason,
+            stored_decoys,
+        }
+    }
+}
+
 /// What the caller wants done about decoys: the whole decoy decision, stated
 /// once, before the file is read.
 ///
@@ -126,9 +172,9 @@ impl std::str::FromStr for DecoyPolicy {
 /// What to do with a peak this reader cannot annotate.
 ///
 /// A kept peak lands at the m/z the file measured, where an annotated one lands
-/// at the m/z its annotation implies. Nothing downstream can tell the two apart
-/// once they are in the arena, so which of them a library is made of is the
-/// caller's decision rather than a fallback the reader picks.
+/// at the m/z its annotation implies. Unknown keys remain recognizable downstream
+/// and cannot establish fragment chemistry. The policy chooses which peaks to
+/// retain; operation eligibility is resolved over all retained labels.
 ///
 /// Asked only of a library that annotates nothing at all, with
 /// [`KeepAll`](Self::KeepAll) the exception that asks nothing. mzPAF spells

rust/timsquery/src/models/query_handle.rs

@@ -300,12 +300,12 @@ mod tests {
             analyte: analyte::Analyte::from_sequence("PEP").as_input(),
             ..Default::default()
         });
-        // Sealed `IfMissing` with no shipped decoy, so the arena derives ±
-        // variants -- and variant 1 is where an ion-annotated label would shift.
+        // Opaque labels disable generation for the whole library.
         let c = c
             .seal(DecoyPolicy::IfMissing)
             .expect("fixture ids are usable");
-        let q = Query::new(&c, c.flat_for(first_row(&c), 1));
+        assert_eq!(c.variants_per_row(), 1);
+        let q = Query::new(&c, c.flat_for(first_row(&c), 0));
         let frags: Vec<_> = q.iter_fragments_refs().collect();
         assert!((frags[0].1 - 300.0).abs() < 1e-9);
     }

rust/timsquery/src/models/target_columns.rs

@@ -6,6 +6,8 @@ use crate::chemistry::analyte::{
 };
 use crate::models::capabilities::{
     DecoyPolicy,
+    DecoyResolution,
+    DecoyResolutionReason,
     DecoyStrategy,
     MASS_SHIFT_VARIANTS,
     TargetCapabilities,
@@ -222,6 +224,7 @@ pub enum TargetBuildError {
 #[derive(Debug, Clone)]
 pub struct TargetColumns<L: KeyLike> {
     pub(crate) caps: TargetCapabilities,
+    decoy_resolution: Option<DecoyResolution>,
     // per-target scalars, len = n_rows; addressed by `RowIdx`, never by a
     // caller-supplied id (those live in `source_ids`)
     pub(crate) precursor_mz: Vec<f64>,
@@ -287,7 +290,10 @@ impl<L: KeyLike> TargetColumnsBuilder<L> {
         self.inner.n_fragments()
     }
 
-    pub fn seal(self, decoys: DecoyPolicy) -> Result<TargetColumns<L>, TargetBuildError> {
+    pub fn seal(self, decoys: DecoyPolicy) -> Result<TargetColumns<L>, TargetBuildError>
+    where
+        L: DecoyShift,
+    {
         self.inner.seal(decoys)
     }
 }
@@ -296,6 +302,7 @@ impl<L: KeyLike> TargetColumns<L> {
     fn empty(caps: TargetCapabilities) -> Self {
         Self {
             caps,
+            decoy_resolution: None,
             precursor_mz: Vec::new(),
             charge: Vec::new(),
             rt_seconds: Vec::new(),
@@ -547,7 +554,10 @@ impl<L: KeyLike> TargetColumns<L> {
         self.frag_off[tgt] as usize..self.frag_off[tgt + 1] as usize
     }
 
-    fn seal(mut self, decoys: DecoyPolicy) -> Result<Self, TargetBuildError> {
+    fn seal(mut self, decoys: DecoyPolicy) -> Result<Self, TargetBuildError>
+    where
+        L: DecoyShift,
+    {
         assert_eq!(self.analytes.len(), self.n_rows());
         assert_eq!(self.entry_names.len(), self.n_rows());
         self.analytes
@@ -565,7 +575,20 @@ impl<L: KeyLike> TargetColumns<L> {
         self.build_decoy_groups()?;
         self.build_source_ids()?;
         let ships_decoys = self.is_decoy.iter().any(|&d| d);
-        self.caps.decoys = decoys.strategy(ships_decoys);
+        let resolution = DecoyResolution::resolve(
+            decoys,
+            stored_decoys,
+            self.frag_labels
+                .iter()
+                .all(|label| label.supports_decoy_shift()),
+        );
+        if resolution.reason == DecoyResolutionReason::MissingFragmentChemistry {
+            tracing::info!(
+                "Mass-shift decoy generation disabled library-wide: missing fragment chemistry; using retained rows only"
+            );
+        }
+        self.caps.decoys = resolution.strategy;
+        self.decoy_resolution = Some(resolution);
         // Nothing to build when no groups were declared: a row is then its own
         // group, which `decoy_group_code` derives. Only the case that actually
         // loses information is worth a word.
@@ -585,6 +608,13 @@ impl<L: KeyLike> TargetColumns<L> {
         Ok(self)
     }
 
+    /// The decision made during sealing, including why generation was disabled.
+    pub fn decoy_resolution(&self) -> &DecoyResolution {
+        self.decoy_resolution
+            .as_ref()
+            .expect("sealed arenas have a decoy resolution")
+    }
+
     pub fn capabilities(&self) -> &TargetCapabilities {
         &self.caps
     }

rust/timsquery/src/serde/library_file.rs

@@ -178,14 +178,15 @@ impl ElutionGroupCollection {
 /// `geom.frag_labels`/`geom.frag_mzs` (same length). The columnar store itself
 /// stores query geometry; readers that supply reference intensities populate
 /// the sidecar for scoring. Geometry-only `Mzpaf` inputs leave it `None`;
-/// the `Str` variant has no intensity sidecar. Extraction ignores intensities.
+/// both variants can carry the sidecar. Target-list JSON supplies none. Extraction ignores intensities.
 pub enum TargetTable {
     Mzpaf {
         geom: TargetColumns<IonAnnot>,
         frag_intens: Option<Vec<f32>>,
     },
     Str {
         geom: TargetColumns<Arc<str>>,
+        frag_intens: Option<Vec<f32>>,
     },
 }
 
@@ -339,7 +340,10 @@ impl TargetTable {
                     });
                 }
                 let geom = geom.seal(DecoyPolicy::Never)?;
-                Ok(TargetTable::Str { geom })
+                Ok(TargetTable::Str {
+                    geom,
+                    frag_intens: None,
+                })
             }
         }
     }
@@ -578,7 +582,7 @@ mod tests {
         }
         match super::read_targets(path).expect("fixture loads") {
             super::TargetTable::Mzpaf { geom, .. } => ids(&geom),
-            super::TargetTable::Str { geom } => ids(&geom),
+            super::TargetTable::Str { geom, .. } => ids(&geom),
         }
     }
 

rust/timsquery/src/traits/fragment_label.rs

@@ -9,6 +9,12 @@ use std::sync::Arc;
 /// label-generic flyweight compiles. Labels without ion chemistry apply the
 /// identity shift.
 pub trait DecoyShift {
+    /// Whether this label supplies the facts required for a mass-shift decoy.
+    /// The arena requires this for every retained fragment before generating any.
+    fn supports_decoy_shift(&self) -> bool {
+        false
+    }
+
     /// Decoy m/z shift. Identity for labels without ion chemistry.
     fn decoy_shift_mz(&self, mz: f64, shift: f64) -> f64;
 }
@@ -31,6 +37,13 @@ pub trait FragmentLabel: KeyLike + DecoyShift {
 }
 
 impl DecoyShift for IonAnnot {
+    fn supports_decoy_shift(&self) -> bool {
+        !matches!(
+            self.series_ordinal(),
+            micromzpaf::IonSeriesOrdinal::unknown { .. }
+        ) && self.get_charge() > 0
+    }
+
     fn decoy_shift_mz(&self, mz: f64, shift: f64) -> f64 {
         // Verbatim `create_mass_shifted_decoy` rule: shift ordinal>2 (and
         // ordinal-less) fragments by `shift / charge`; leave ordinal<=2 alone.

rust/timsquery_cli/src/commands.rs

@@ -94,7 +94,7 @@ pub fn main_query_index(args: QueryIndexArgs) -> Result<(), CliError> {
             &put_path,
             batch_size,
         ),
-        TargetTable::Str { geom } => stream_process_batches(
+        TargetTable::Str { geom, .. } => stream_process_batches(
             &geom,
             aggregator_use,
             &index,
@@ -493,7 +493,7 @@ mod tests {
         let arena = read_query_elution_groups(tmp_file.path()).unwrap();
 
         let geom = match arena {
-            TargetTable::Str { geom } => geom,
+            TargetTable::Str { geom, .. } => geom,
             TargetTable::Mzpaf { .. } => {
                 panic!("string labels must land in the Str arena, not Mzpaf")
             }