Score sequence-free libraries with library-wide chemistry fallbacks

[jspaezp]

Sequence-free libraries can now reach common spectrum scoring without treating unknown fragment keys as chemical annotations. A library with unannotated peaks and no decoys automatically uses raw scoring; it no longer generates charge-assumed mass-shift decoys or attempts FDR estimation.

For example, an mzSpecLib spectrum containing precursor m/z/charge and reference peaks, but no sequence and only ? peak annotations, now yields raw spectrum scores with nullable sequence and no q-value column. Programmatic TargetTable::Str libraries can also supply reference intensities; even a string key "y3" remains opaque.

• Resolve generated-decoy eligibility at arena sealing, across all retained labels. Existing supplied decoys remain available under if-missing/never.
• Gate fragment-isotope extraction library-wide. Split its score block and use derive-generated names to select matching feature values for LDA, hybrid, GBM, MLP and dashboard. Disabled model features are omitted; disabled raw score fields are NaN.
• With decoys, retain calibrated scoring/rescoring. Without decoys, score over full acquisition RT and bypass calibration, competition, rescoring and q-value filtering. Parquet records result_mode=raw; the run report records per-file raw_scores=true and the library-wide scoring_plan (plus calibration_scoring_plan for a separate calibration library). Plans include the requested decoy policy, resolved strategy, stored-decoy count and retained resolution reason. The report borrows the same plans used by scoring/Parquet, without copying isotope buffers.
• Validate string-labelled intensity sidecars and preserve their source labels, IDs, groups and chemistry while assigning packed unknown extraction keys. This bridge currently accepts at most 255 opaque peaks per entry and rejects larger entries. Target-list JSON schemas still supply no intensities and remain extraction-only.

Validation: 452 unit tests passed (one additional ignored), eight query contract tests, workspace Clippy with dashboard, serial CLI check, and task fmt.

Real-data checks use the supplied 948,957-spectrum / 27,287,703-peak HeLa library and Astral mzML:

Library	Outcome
Original, merged-main baseline	28,512 targets at 1% FDR
No analyte chemistry; fragment annotations and supplied decoys retained	28,116 targets at 1% FDR
No analyte chemistry or fragment annotations; supplied decoys retained	27,972 targets at 1% FDR
1,000 target-only entries, no chemistry/annotations	537 raw scored candidates; no q-value column

Degraded output sequences are null; IDs/names survive. The sequence-only fixture uses explicit 0@ annotations so fragments do not reference a deleted analyte; it retains fragment-isotope scoring. Fixtures/results are local and gitignored; the original library/raw file are untouched.

Concrete mzSpecLib degradation example

This is spectrum 1 from the original HeLa library (entry body; the unchanged library header is omitted):

<Spectrum=1>
MS:1003061|library spectrum name=AAAAC[UNIMOD:4]LDK/2
[3]MS:1003275|other attribute name=msspeculator:decoy_pair_id
[3]MS:1003276|other attribute value=10
MS:1000041|charge state=2
MS:1000744|selected ion m/z=410.20509552
[1]MS:1000896|normalized retention time=22.740475
[1]UO:0000000|unit=UO:0000031|minute
MS:1002815|inverse reduced ion mobility=1.15999700
MS:1003059|number of peaks=10
<Analyte=1>
MS:1003270|proforma peptidoform ion notation=AAAAC[UNIMOD:4]LDK/2
MS:1000888|stripped peptide sequence=AAAACLDK
MS:1003043|number of residues=8
MS:1003637|theoretical neutral monoisotopic mass=818.39563811
MS:1000885|protein accession=sp|Q9H3U1|UN45A_HUMAN
<Peaks>
143.081504	1.000000	b2
147.112804	0.044033	y1
214.118618	0.204782	b3
262.139747	0.047158	y2
285.155732	0.045861	b4
375.223811	0.024744	y3
535.254459	0.079902	y4
606.291573	0.164994	y5
677.328687	0.690483	y6
748.365801	0.127760	y7

Case	Removed	Rewritten	Preserved
Baseline	Nothing	Nothing	Original entry
No analyte chemistry (no-analytes)	Entire <Analyte=1> block: ProForma ion notation (MS:1003270), stripped sequence (MS:1000888), residue count (MS:1003043), theoretical neutral mass (MS:1003637), protein accession (MS:1000885)	Each peak annotation becomes unassociated: b2 → 0@b2, y1 → 0@y1, etc. This removes the implicit reference to deleted analyte 1 while retaining ion series/ordinal/charge information.	Every spectrum-level field above, all ten peak m/z/intensity pairs and their order
No analyte chemistry or fragment annotations (no-chemistry)	Same entire Analyte block, plus all peak annotation information	Third peak column becomes ? for every peak	Same spectrum metadata and all ten peak m/z/intensity pairs/order
Target-only raw test (raw-1000)	Same chemistry/annotation removal as no-chemistry; additionally exclude decoy spectra and take the first 1,000 retained targets	Same ? peak annotations	Each retained target's identity, pairing field, coordinates and intensities

For example, the first peak changes as follows (columns are m/z, reference intensity, annotation):

baseline:      143.081504  1.000000  b2
no-analytes:   143.081504  1.000000  0@b2
no-chemistry:  143.081504  1.000000  ?

MS:1003061|library spectrum name=AAAAC[UNIMOD:4]LDK/2 remains unchanged in every case. It is an opaque entry name, not a source of sequence or charge; charge remains separately supplied by MS:1000041. The full-library degradations also retain all supplied decoy spectra, their decoy declarations and pairing IDs. Header provenance remains unchanged. This entry supplies no separate molecular-formula field to remove.

Spectrum-name hash stress test

Replaced only MS:1003061 values with sha256(original_name.encode("utf-8")).hexdigest() in the fully degraded library and the target-only raw fixture. Example:

MS:1003061|library spectrum name=d521045bbfd3b4eb0e66c120df5648f368f655bad5fbe76e8e07aacefcdaddcd

This hashes AAAAC[UNIMOD:4]LDK/2 without its trailing newline. All 948,957 names became distinct hashes. A line-by-line comparison verified every other input line unchanged: spectrum keys, charge, coordinates, peaks/intensities, decoy declarations, pairing IDs and header provenance.

• Raw test: the same 537 candidates survive from 1,000 entries. Every non-identity output value is identical, including charge and raw scores; sequences remain null and scoring plans match. Source IDs/names become hashes. Singleton group IDs also follow those source IDs, without changing membership.
• Full supervised test: the fitted calibration curve, calibrant point set, tolerances and scoring plans are identical (saved point order differs, with tiny rounding differences in residual mean/stdev summaries); both runs score 519,932 entries and retain 262,201 candidates after competition. Output sequences remain null. Among the 67,387 entries shared by the q-filtered outputs, every column except names/source IDs, discriminant_score and qvalue is identical, including charge and declared group IDs.
• Identity still affects rescoring: canonicalize_and_shuffle sorts by source ID before seeded shuffling into folds. mzSpecLib spectrum names supply those IDs, so hashing changes fold assignments. The 1% FDR count changes 27,972 → 28,137; q-filtered outputs contain 103,481 versus 102,332 rows (36,094 original-only, 34,945 hashed-only). This test establishes chemistry/charge independence from names, not invariance of learned scores to renamed identities.

Local generators, line-verification manifests and result comparisons are in .scratch/97-sequence-free/; no source library or production code changed for this test.

This does not validate decoy strategies for new analyte classes or add arbitrary adduct/negative-ion handling. force retains its existing reader behavior of dropping supplied decoys before sealing; if generation is then unavailable, the retained rows use raw scoring. Supervised calibration retains its existing RT-input contract.

Closes #97; follows #142, #144 and #145. Sidecar ownership cleanup in #118 remains separate.

[github-actions]

Apex-finder bench

cargo run -p apex_sim --release --example bench -- 1000 2

Sensitivity + timing across canonical scenarios

=== summary (n=1000, tol=±2 cycles) ===
  scenario                      pass2%  pass1%   medErr    us/run
  clean                          100.0   100.0        0     16.16
  moderate_noise                  92.7    92.7        0     15.04
  high_noise+interference         59.3    59.3        1     16.22
  heavy_interference               7.4     7.4       88     14.81
  mismatched_library              59.8    59.8        1     14.72
  absent_top_fragment             16.0    16.0       65     15.41
  absent_precursor                59.3    59.3        1     15.11

=== broad apex-finding (n=500, tol=±2 cycles) ===
  scenario                      pass2%  pass1%   medErr    us/run
  broad_clean                    100.0   100.0        0     97.71
  broad_moderate_noise            50.6    50.6        1     93.24
  broad_high_noise+interf         27.6    27.6      308     93.31
  broad_hard_3x_density            0.8     0.8      487     94.02
  broad_mismatched_library        41.0    41.0      161     94.40
  broad_measured_density          49.6    49.6       13    121.33

=== narrow recovery (n=1000, tol=±2 cycles) ===
  scenario                      pass2%  pass1%   medErr    us/run
  narrow_clean                   100.0   100.0        0     10.73
  narrow_moderate_noise           81.3    81.3        0     10.06
  narrow_high_noise+interf        65.8    65.8        1      9.96
  narrow_hard_3x_density          10.9    10.9       34     10.02
  narrow_mismatched_library       62.8    62.8        1      9.91
  narrow_measured_density         83.6    83.6        0     12.29

=== narrow score discrimination (AUC, n_seed_pairs=1000) ===
  scenario                         AUC   med+signal    med-noise
  narrow_clean                   1.000     4.329e14      7.199e7
  narrow_moderate_noise          0.870      4.054e9      3.054e8
  narrow_high_noise+interf       0.830      2.761e9      3.602e8
  narrow_hard_3x_density         0.684      3.644e9      1.244e9
  narrow_mismatched_library      0.842      2.172e9      3.575e8
  narrow_measured_density        0.858      4.341e9      2.562e8

commit 7ff2817

[Copilot]

🟡 Changes recommended

Moderate issues remain in raw RT semantics, empty-arena decoy handling, and Phase 6 timing.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR adds sequence-free library scoring with chemistry fallbacks, raw-score output, and opaque string-label support.

Changes:

• Gates isotope features and mass-shift decoys library-wide.
• Adds raw scoring without calibration, rescoring, competition, or q-values.
• Supports string-labelled libraries with optional intensity sidecars.

File summaries

File	Summary
rust/timsseek/src/scoring/timings.rs	Adds raw-score performance reporting.
rust/timsseek/src/scoring/plan.rs	Resolves isotope capability and feature selection.
rust/timsseek/src/scoring/pipeline.rs	Supports raw extraction and optional calibration.
rust/timsseek/src/scoring/parquet_writer.rs	Writes raw-mode schemas and metadata.
rust/timsseek/src/scoring/blocks/lazy.rs	Separates isotope score fields.
rust/timsseek/src/ml/qvalues.rs	Applies scoring-plan feature projections.
rust/timsseek/src/data_sources/reference_library.rs	Supports opaque labels and intensity sidecars.
rust/timsseek_cli/tests/build_library.rs	Updates target-table matching tests.
rust/timsseek_cli/src/processing.rs	Adds raw execution and decoy-free handling.
rust/timsquery/src/traits/fragment_label.rs	Defines fragment and decoy-shift capabilities.
rust/timsquery/src/serde/library_file.rs	Supports string-label intensity sidecars.
rust/timsquery/src/models/target_columns.rs	Gates generated decoys by library capability.
rust/timsquery/src/models/query_handle.rs	Updates opaque-label behavior.
rust/timsquery/src/models/capabilities.rs	Clarifies capability semantics.
rust/timsquery_cli/src/commands.rs	Handles extended table variants.
docs/development.md	Documents fallback and raw scoring modes.

Review details

Suppressed comments (1)

rust/timsseek/src/scoring/pipeline.rs:744

• The raw branch now sets calibrated_rt_seconds to NaN, but CandidateMetadata still documents this field as equal to library_rt when no calibration is applied. Since Rt::compute also derives calibrated_delta_rt from that value, update the public contract to distinguish raw mode's unavailable, unit-agnostic calibration rather than leaving consumers with contradictory semantics.

                            source_id: identity.source_id,
                            library_rt: query.rt_seconds(),
                            calibrated_rt_seconds: f32::NAN,
                            ref_mobility_ook0: query.mobility_ook0(),
                            ref_precursor_mz: query.mono_precursor_mz(),

• Files reviewed: 16/16 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.