Skip to content

perf(decoder): add bounded columnar decoding - #4

Merged
muthmann merged 4 commits into
mainfrom
agent/optimize-decoding-performance
Aug 12, 2026
Merged

perf(decoder): add bounded columnar decoding#4
muthmann merged 4 commits into
mainfrom
agent/optimize-decoding-performance

Conversation

@muthmann

@muthmann muthmann commented Aug 12, 2026

Copy link
Copy Markdown
Owner

Summary

  • decode EVT3 data directly into columnar storage and stable NumPy arrays
  • add bounded-memory RAW and HDF5 file readers, including trigger-aware Python batches
  • stream CLI CSV and binary output without materializing the full recording
  • optimize sparse vector masks, byte conversion, capacity planning, GIL use, and release profiles
  • add Criterion coverage, a PGO build workflow, measured optimization results, and updated README/feature documentation
  • include the NumPy-native Events and Augur ingress foundation required by the optimized Python workflow
  • retain and strengthen the C++ reference comparison with a reproducible like-for-like CSV benchmark

User impact

Existing decode_file, decode_file_with_triggers, and decode_bytes calls remain compatible and use the optimized core automatically. Large-file users can opt into decode_file_batches; users who need external trigger edges can use decode_file_batches_with_triggers, which yields (Events, TriggerEvents) per bounded batch.

Measured on test_data/laser.raw:

  • Python full-file path: 1.82x faster and 59.1% lower maximum RSS than the previous implementation
  • bounded Python path: 92.1% lower maximum RSS than the previous full-memory path
  • streaming CLI: 1.67x faster and 93.2% lower maximum RSS than the previous materialized CLI
  • sparse vector-mask benchmark: about 1.89x throughput
  • like-for-like full CSV: Rust 7.414 s versus optimized C++ 12.028 s, or 1.62x faster
  • instrumented CSV run: Rust 63.3 MB RSS versus C++ 28.3 MB; C++ remains more memory-efficient in this comparison

The Rust and C++ CSV outputs were byte-identical on an 8-MiB input prefix. The benchmark runner now verifies this and alternates measured run order.

Validation

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace
  • HDF5 Clippy and full HDF5 test suite
  • release Python extension build
  • Python test suite: 21 passed
  • reproducible Rust/C++ comparison mode executed successfully
  • byte-identical Rust/C++ CSV prefix with SHA-256 398d63a52eeb7291caa1346037f674ef0b6209772d3c512e2aa6c70b9aed12f4
  • exact aggregate checksum comparison for the real-file Python result
  • PGO build and output comparison
  • local Markdown-link audit

Real ECF fixture checks were skipped because the external ECF plugin is not installed. Synthetic HDF5 and HDF5 batch tests passed.

@muthmann
muthmann marked this pull request as ready for review August 12, 2026 13:22
@muthmann
muthmann merged commit a425f8c into main Aug 12, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant