HATS Phase‑2: lazy partition-aware dataset with bundle cache, docs, and tests

[mtauraso]

Motivation

• Replace eager full-catalog materialization with a lazy, partition-aware access model to support large HATS catalogs.
• Improve random/clustered read performance by introducing a bounded row-bundle cache and column projection.
• Provide a clear design and usage examples to guide implementation and adoption.

Description

• Add lazy index/cache/accessor components: HATSPartitionIndex, HATSRowBundleCache, and HATSLazyAccessor implementing global index mapping, LRU row-bundle caching, and partition-aware reads.
• Refactor HyraxHATSDataset to open HATS catalogs lazily via lsdb.open_catalog/lsdb.read_hats, extract per-dataset hats config, compute __len__ from the partition index, and route per-column get_<column> getters to the lazy accessor.
• Implement safe fallbacks for catalogs without partition metadata and conservative full-catalog compute when necessary, and add column-projection logic and required-column detection from the data_request.
• Add design doc specs/hasts_dataset_phase2.md, example notebook docs/notebooks/hats_dataset_phase2_loopback.ipynb, and unit tests in tests/hyrax/test_hats_dataset_phase2.py covering index resolution, catalog open fallback, and cache behavior.

Testing

• Added unit tests in tests/hyrax/test_hats_dataset_phase2.py that validate HATSPartitionIndex mapping, preference for lsdb.open_catalog with fallback to read_hats, and basic bundle-cache hit/miss behavior.
• These tests exercise the fake-catalog abstractions and the new dataset accessor logic.
• No automated test execution or CI results were included in this rollout; test execution should be performed in CI or locally (e.g., pytest tests/hyrax/test_hats_dataset_phase2.py).

---

Codex Task

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB