Use nvCOMP Gzip decoding for FITS tiles by tpn · Pull Request #76 · NVIDIA/cuPhoton

tpn · 2026-09-29T22:10:03Z

FITS tile loading now validates common fixed Gzip headers in batches and passes intact streams to nvCOMP. An isolated two-file comparison measured full-reader time of 73.71 → 63.11 ms (14.4% lower), with identical decoded bytes. Most of that gain comes from host-side header validation; resident Gzip and aligned DEFLATE decoding took similar time.

gzip_decoder="auto" selects native Gzip when available. Python APIs, FITS descriptors and --xdr-gzip-decoder also accept gzip to require it or deflate to select raw DEFLATE. Raw payloads are aligned on the GPU; older extensions remain usable in automatic mode. Decode failures propagate.

Stacked on #74. Validation includes byte-for-byte Astropy parity, optional-header and alignment checks, CPU CLI/report coverage, native builds and type checks. The timing above measures the combined adapter and header-validation change; it is not a codec-only speedup.

copy-pr-bot · 2026-09-29T22:10:06Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

coderabbitai · 2026-09-29T22:10:11Z

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)

^codex/013-.*$

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: NVIDIA/cuPhoton/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: abcc9db9-1aa7-42c6-a7db-cb05928ef49a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: NVIDIA/cuPhoton/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d1b9b309-5d12-4ddb-9f1d-c7c622ccd218

📥 Commits

Reviewing files that changed from the base of the PR and between a0f728e and 0bdabba.

📒 Files selected for processing (15)

docs/components/xdr.md
src/cuphoton/core/cli/fits.py
src/cuphoton/core/fits_options.py
src/cuphoton/xdr/benchmark_fits.py
src/cuphoton/xdr/convenience.py
src/cuphoton/xdr/nvcomp_batch.py
src/cuphoton/xdr/prefetch.py
src/cuphoton/xdr/reader.py
src/cuphoton/xdr/src/nvcomp_batch_ext.cpp
tests/core/test_cli_contract.py
tests/core/test_fits_io.py
tests/xdr/test_benchmark_fits.py
tests/xdr/test_frontend.py
tests/xdr/test_gzip_gpu.py
tests/xdr/test_nvcomp_batch.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.

📝 Walkthrough

Walkthrough

The xDR decoding pipeline now supports auto, gzip, and deflate decoder choices. The option flows through reader, batch, benchmark, and CLI interfaces. The native extension adds batched Gzip decompression alongside DEFLATE.

Changes

xDR Gzip Decoding

Layer / File(s)	Summary
Native Gzip and DEFLATE decompression `src/cuphoton/xdr/src/nvcomp_batch_ext.cpp`	The extension uses shared decompression logic for Gzip and DEFLATE. It adds stream-ordered scratch allocation and a `batch_gzip_decompress` binding.
Python decoder selection and input handling `src/cuphoton/xdr/nvcomp_batch.py`, `tests/xdr/test_nvcomp_batch.py`, `tests/xdr/test_gzip_gpu.py`	The batch decoder selects native Gzip or DEFLATE, aligns raw DEFLATE inputs when needed, and retains input and codec resources. Tests cover decoder selection, gzip headers, alignment, stream behavior, and codec lifecycle.
Decoder option and xDR integration `src/cuphoton/core/fits_options.py`, `src/cuphoton/core/cli/fits.py`, `src/cuphoton/xdr/{reader,prefetch,convenience,benchmark_fits}.py`, `docs/components/xdr.md`, `tests/core/*`, `tests/xdr/{test_frontend,test_benchmark_fits}.py`	The decoder option is validated and forwarded through xDR readers, batch APIs, benchmarks, and the CLI. Tests check forwarding and defaults. The documentation describes the choices and their behavior.

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to 0bdab

The decoder selection and forwarding changes have no established merge-blocking issue. Merge after normal build and test checks.

🚥 Pre-merge checks | ✅ 2

✅ Passed checks (2 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{Comment @coderabbitai help to get the list of available commands.}

tpn · 2026-09-30T03:22:39Z

@coderabbitai review

coderabbitai · 2026-09-30T03:22:48Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

melo-gonzo

All 459 selected decoder/FITS tests passed on RTX 6000 Ada using this PR’s CI-built native extension, but the exception cleanup below needs fixing before approval. Recommend landing after 0.1.3; the ARM wheel check also needs rerunning after its CFITSIO download timeout.

melo-gonzo · 2026-09-30T03:59:09Z

        else cp.empty(total, dtype=cp.uint8)
    )
+    if aligned_owners and keepalive is None:
+        d_out = _retain_decode_inputs(d_out, aligned_owners)


issue: With keepalive=None, an allocation or owner-wrapping failure after _align_deflate_inputs queues its kernel can release the input and repack buffers while GPU work is still pending. Owners must survive until that work completes, including on exceptions. Preserve #81’s exception-time synchronization around repacking and subsequent allocation/decode, including partial metadata-table construction, and add a failure-injection regression test. Also retain #81’s exception-safe pooled capsule construction when combining the native changes.

Signed-off-by: Trent Nelson <trentn@nvidia.com>

Align raw DEFLATE payloads on the device before decoding and retain those buffers through asynchronous work. Keep automatic selection compatible with older native extensions. Signed-off-by: Trent Nelson <trentn@nvidia.com>

Signed-off-by: Trent Nelson <trentn@nvidia.com>

Bind Python DEFLATE decoding to the current device and CUDA stream. Retain that stream until its cached codec is destroyed, ordering decode after compressed-input uploads and alignment copies. Keep Gzip dispatch and alignment covered by CPU checks and a delayed producer GPU regression. Signed-off-by: Trent Nelson <trentn@nvidia.com>

This was referenced Sep 29, 2026

Select compatible hardware for Gzip decompression #77

Open

Fuse FITS postprocessing with runtime CLI and API controls #74

Open

tpn force-pushed the codex/compression-gzip-20260929 branch from ad75c65 to edc5b80 Compare September 30, 2026 01:55

tpn changed the base branch from main to codex/compression-xdr-20260929 September 30, 2026 01:55

tpn mentioned this pull request Sep 30, 2026

Fix xDR Deflate alignment and nvCOMP output buffers #81

Merged

tpn force-pushed the codex/compression-gzip-20260929 branch from edc5b80 to 0bdabba Compare September 30, 2026 02:51

tpn marked this pull request as ready for review September 30, 2026 03:22

tpn requested a review from melo-gonzo as a code owner September 30, 2026 03:22

tpn added the ai-review Request a focused CodeRabbit review label Sep 30, 2026

melo-gonzo requested changes Sep 30, 2026

View reviewed changes

tpn added 4 commits September 30, 2026 21:44

Decode FITS gzip tiles without stripping their framing

d351b96

Signed-off-by: Trent Nelson <trentn@nvidia.com>

Allow explicit selection of FITS gzip decoders

fac8206

Align raw DEFLATE payloads on the device before decoding and retain those buffers through asynchronous work. Keep automatic selection compatible with older native extensions. Signed-off-by: Trent Nelson <trentn@nvidia.com>

Expose gzip decoder choice through FITS APIs and CLI

dbac763

Signed-off-by: Trent Nelson <trentn@nvidia.com>

tpn force-pushed the codex/compression-xdr-20260929 branch from a0f728e to 3c7e2c0 Compare October 1, 2026 05:05

tpn force-pushed the codex/compression-gzip-20260929 branch from 0bdabba to e33dc4d Compare October 1, 2026 05:05

Navigation Menu

Use nvCOMP Gzip decoding for FITS tiles - #76

Use nvCOMP Gzip decoding for FITS tiles#76
tpn wants to merge 4 commits into
codex/compression-xdr-20260929from
codex/compression-gzip-20260929

tpn commented Sep 29, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Sep 29, 2026

Uh oh!

coderabbitai Bot commented Sep 29, 2026 •

edited

Loading

Review skipped

Walkthrough

Changes

Uh oh!

tpn commented Sep 30, 2026

Uh oh!

coderabbitai Bot commented Sep 30, 2026 •

edited

Loading

Uh oh!

melo-gonzo left a comment

Uh oh!

melo-gonzo Sep 30, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

tpn commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-bot Bot commented Sep 29, 2026

Uh oh!

coderabbitai Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Walkthrough

Changes

Uh oh!

tpn commented Sep 30, 2026

Uh oh!

coderabbitai Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

melo-gonzo left a comment

Choose a reason for hiding this comment

Uh oh!

melo-gonzo Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tpn commented Sep 29, 2026 •

edited

Loading

coderabbitai Bot commented Sep 29, 2026 •

edited

Loading

coderabbitai Bot commented Sep 30, 2026 •

edited

Loading