Skip to content

OpenEXR: read PIZ-compressed files (Poly Haven EXRs) - #887

Merged
paul-hammant merged 7 commits into
mainfrom
feat/exr-piz
Oct 30, 2026
Merged

paul-hammant merged 7 commits into
mainfrom
feat/exr-piz

Conversation

@paul-hammant

Copy link
Copy Markdown
Collaborator

OpenEXR PIZ for ae3d.openexr, as Nic asked (875-1). Every Poly Haven EXR is PIZ, so engine_set_environment / AE3D_ENV could not load any of them; they do now. Pure Aether (ae3d.exrpiz, ~480 lines), no C helper.

Format coverage

compression
NONE, RLE, ZIPS, ZIP read (unchanged)
PIZ read: HALF and FLOAT channels, RGB, RGBA, lone Y, any data window, lines in any order, blocks the writer stored raw
PXR24, B44, B44A, DWAA, DWAB, tiled, deep, multipart still refused by name (header-only fixtures coded_{pxr24,b44,dwaa}.exr)

The change in ae3d.openexr is a 3-line branch plus the refusal threshold; half_to_float now keeps a half's -0 as -0 (it returned +0), needed for bit equality.

How equality to OpenEXR was established

Official OpenEXR 3.5.2 (brew, reference only; nothing linked into ae3d or built by ci) via tools/exr_reference/exrref.cpp: write makes PIZ files, dump prints two position-sensitive hashes of the library's decode (RGBA float32 bits). test_envimage computes the same hashes over ae3d.openexr's decode and compares: 25 small PIZ files + the 2 Poly Haven assets, bit for bit. Also 250 randomly sized/patterned/channelled files (sizes 1..1000, offset windows, all patterns, HALF/FLOAT/Y) compared to the library's decode: 0 differ.

Covered: RGB, RGBA, Y; HALF and FLOAT (the library does write PIZ FLOAT); widths 1, 4, 37, 1000; heights 1, 32, 33, 45, 64; single scanline; flat and all-zero images (degenerate tables); data window off the origin; both wavelet forms (14-bit below 16384 distinct values, 16-bit above, via a bit-pattern ramp: the first version of the 16-bit step was wrong and only that fixture caught it; the Poly Haven files never reach it); long (>14-bit) Huffman codes; run codes of zero and nonzero words; blocks the library stored raw. tests/fixtures/env/PIZ.md tabulates each fixture; tools/exr_reference/blocks.py says per block whether a file is really PIZ or stored raw (small and noisy files are raw; I found that out the hard way and replaced fixtures that were silently raw).

Negative controls (each fails test_envimage)

wavelet one level short / one level too many; value table reversed; block height 16 not 32; one Huffman code-length bit flipped; canonical codes assigned shortest-first; codes over 14 bits not decoded; run code repeats one word too few; 14-bit form for every range; 16-bit form for every range; 14-bit odd-bit term dropped; 16-bit difference halved the wrong way; 16-bit half-range offset dropped; FLOAT read as one plane; bitmap placed at offset 0; lines laid wrong; -0 turned back to +0; Huffman count check dropped. 18 of 18 caught (the first pass caught 15 of 18 and led to the ramp fixture and the damaged-bit-count checks).

Costs (M6, release)

decode PIZ ZIP
1k Poly Haven (FLOAT RGB, 1.3 MB) 22-23 ms --
1024x512 FLOAT RGB, same synthetic image 24 ms 22 ms
4096x2048 HALF RGB 238 ms (21.6 MB) 295 ms (23.6 MB)
4096x2048 FLOAT RGB (half-precision values) 362 ms (21.2 MB) 322 ms (6.7 MB)

measure_environment on the 4k FLOAT file: set_environment 357/388 ms (GL/Vulkan) PIZ against 355/376 ZIP; first frame and per-frame cost unchanged.

Verification

  • Engine: both Poly Haven assets through engine_set_environment in test_environment (renderer irradiance onto 4 normals = decoded integral to 10%, both backends); test_backend_parity draws one with GL and Vulkan: 0 of 147,456 channels differ at AE3D_MSAA=0, 719 at 4x (edges; the gradient case has 681).
  • Khronos validation with synchronization validation (VK_LAYER_ENABLES=...SYNCHRONIZATION_VALIDATION_EXT, VUID-/SYNC-HAZARD- counted): 0 over test_backend_parity, test_environment, street_drive and zombie_city with a Poly Haven map, at 1x and 4x.
  • frames_against 9d2ecd2: 71 frames, 0 differ; also with AE3D_MSL=1 MVK_CONFIG_FAST_MATH_ENABLED=0 both sides: 0 differ.
  • Robustness: cut at every length and every byte changed (to a sequence, to 0 and to 255) over six small PIZ files; bit counts of the Huffman stream damaged. Clean under leaks (0 leaks) and Guard Malloc (libgmalloc defaults to 16-byte alignment, so small overruns can slip through).
  • Full ./ci.sh green (8 skipped: DLSS, ray query, one leak report, as on this machine before).
  • Linux, aether-linux-x64: lane_some gcc+clang -O2/-O0, 140 builds, 0 failed, 0 warnings (another agent's lane shared /tmp log dirs during the run, so the warning grep may have missed some; the exit codes are sound); test_envimage passes (170 checks) built with gcc and clang on both aether-linux-x64 and aether-linux (arm64), no warnings.

Fixtures and licences

tests/fixtures/env/polyhaven/: abandoned_tank_farm_05_1k.exr and aerodynamics_workshop_1k.exr, 1k PIZ FLOAT, unchanged from polyhaven.com, CC0; URLs in tests/fixtures/env/PIZ.md. The OpenEXR library is BSD-3-Clause; the decoder follows its structure (wavelet level loop and Haar steps, code-length table run codes, canonical code assignment), so its notice is in the ae3d.exrpiz header and THIRD_PARTY_LICENSES.md.

Not done / notes

  • Tiled, PXR24, B44, DWA stay refused. A Huffman code over 56 bits is refused (would need >2^32 samples).
  • Replaced coded_piz.exr (header only) with header-only PXR24/B44/DWAA refusals; test_environment now uses coded_b44.exr as its not-a-picture file (one line, may overlap the 2:1 refusal PR).
  • Questions for Nic: ok to keep tools/exr_reference/exrref.cpp (+ blocks.py) in the tree as a dev-only tool? Four 4k-class timings were measured locally only; no 4k file is committed.

🤖 Generated with Claude Code

paul-hammant and others added 6 commits October 28, 2026 22:43
A pure-Aether PIZ block decoder. Not wired in yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PXR24, B44, B44A, DWAA and DWAB are still refused by name. half_to_float
returned +0 for 0x8000 (0 - 0), so a decode could not equal the OpenEXR
library's bit for bit; it is negated by a product now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
25 small PIZ files plus two Poly Haven 1k assets (CC0): hashes of the OpenEXR
library's decode in test_envimage, the fuzz loops over the small ones, damaged
bit counts. tools/exr_reference is the development tool that wrote them (not
built, not linked). coded_piz.exr (a header only) is replaced by header-only
PXR24, B44 and DWAA refusals.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_environment holds the renderer's irradiance for two assets to the decoded
picture's integral on OpenGL and Vulkan; test_backend_parity draws one with
both and holds the frames equal (0 of 147456 channels at one sample).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/test_environment.ae: both sides added checks after ground(); keep the
Poly Haven PIZ assets (this branch) and the environment-map yaw checks (#888).
test_envimage 170/170 and test_environment pass on both backends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
paul-hammant added a commit that referenced this pull request Oct 30, 2026
#886 and #887's linux jobs were cut off at 25 minutes in their last step, the
benchmarks, after merging main rebuilt most of the shard. That job takes 10
to 24 minutes on recent pull requests, and #888's first run also stopped at
25. The change to ci.yml starts a run on main, which saves the cache pull
requests restore.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
paul-hammant added a commit that referenced this pull request Oct 30, 2026
Combined #886 + #887: Nic's answers and EXR PIZ, pre-merged
@paul-hammant
paul-hammant merged commit 58754f8 into main Oct 30, 2026
0 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant