diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 66f5d3b..e5938f7 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -11,7 +11,10 @@ concurrency: cancel-in-progress: true env: - RUST_VERSION: "1.85" + # Must satisfy native/ex_codecs_native/Cargo.toml `rust-version`. + # structured-zstd calls the now-safe __cpuid without an unsafe block on + # x86_64, which needs rustc >= 1.94 (its own 1.92 MSRV metadata is wrong). + RUST_VERSION: "1.94" jobs: test: diff --git a/.gitignore b/.gitignore index a44daf2..712c574 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,6 @@ # The directory Mix will write compiled artifacts to. /_build/ -/prompts/ +/baoulo/ # If you run "mix test --cover", coverage assets end up here. /cover/ @@ -25,10 +25,11 @@ ex_codecs-*.tar # Rust artifacts /native/ex_codecs_native/target/ -/native/ex_codecs_native/Cargo.lock # Rustler precompiled cache /priv/native/ # Benchmark results /bench/results/ +.tool-versions +.DS_Store diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..6727bbb --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,119 @@ +# Changelog + +All notable changes to this project will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [0.2.0] - 2026-07-16 + +### Added + +- **`ExCodecs.Spatial`** — new spatial codec category for point clouds and + Gaussian splats (pure Elixir, no rendering). +- Domain types: `Point`, `PointCloud`, `Gaussian`, `GaussianCloud`, `Bounds`, + `Transform`, `Metadata`. +- Format codecs: + - `:ply` — ASCII and binary PLY (XYZ / RGB / RGBA / normals / attributes, + plus Gaussian PLY conventions) + - `:spatial_binary` — compact `EXCP` binary point clouds + - `:gsplat` — simple `GSPL` binary Gaussian format +- Streaming helpers: `ExCodecs.stream_decode/2`, `ExCodecs.stream_encode/2`, + and `ExCodecs.Spatial.Stream` (materializing streams; optional `source: :file | :binary`). +- Spatial category module `ExCodecs.Spatial` (struct↔format); top-level + `ExCodecs.encode/3` / `decode/3` remain **registry binary codecs only** + (original framework shape). `stream_encode` / `stream_decode` on `ExCodecs` + delegate to Spatial for convenience. +- Example datasets under `priv/examples/spatial/`. +- Guide: `guides/understanding_spatial_codecs.md`. +- Wire-format freeze: `docs/spatial_formats.md` (EXCP / GSPL / PLY rules). +- Spatial tutorial Livebook: `livebooks/06_spatial_codecs.livemd`; existing + Livebooks now target v0.2.0 APIs and metadata. +- Shared codec catalog discovery across compression and spatial categories, + including `ExCodecs.available_codecs/1` and `%ExCodecs.Codec{interface: ...}`. +- Registry `unregister/1` and re-registration on registry process restart. +- Error reasons `:io_error`, `:truncated_input`, and `:output_limit_exceeded`. +- Decode option `:max_output_size` (default **256 MiB**) on all compression + codecs — rejects decompression bombs. + +### Changed + +- **Native NIF is pure Rust** — no C compression libraries: + - Zstd via `structured-zstd` (not libzstd / not stock `ruzstd` encoder) + - Bzip2 via `bzip2` + pure-Rust `libbz2-rs-sys` + - Zlib (Blosc2 inner) via `flate2` rust backend + - LZ4 / Snappy unchanged pure crates +- **Zstd `:level`** (1–22) is functional on the pure-Rust encoder (ratios may + differ from C libzstd at the same numeric level). +- **Blosc2**: C-Blosc2-compatible **chunk** format via pure-Rust `blosc2-pure-rs` + (`:blosclz`, `:lz4`, `:lz4hc`, `:zstd`, `:zlib` + shuffle/bitshuffle). + Golden tests against python-blosc2 fixtures. +- Codec `streaming?` metadata set to `false` until real streaming exists. +- Removed unimplemented Zstd `:window_log` option from docs. +- NIF load checked at registration; NIF calls wrapped so `:nif_not_loaded` + becomes `{:error, %ExCodecs.Error{}}` instead of raising. +- Safer PLY header parsing (no raise on malformed element lines / unknown types). +- Point attributes normalized to **string keys** (atoms accepted at `Point.new/4`). +- Public API docs cover spatial formats alongside compression codecs. +- Spatial encode/decode resolves PLY, EXCP, and GSPL implementations through + the shared ETS catalog while retaining the category-safe Spatial API. +- Test coverage threshold restored to **95%** after expanding spatial and NIF + edge-case coverage. + +### Notes + +- Precompiled NIF checksums must be regenerated when publishing GitHub release + artifacts for `0.2.0`; until then local builds use `force_build` / source compile. +- `native/ex_codecs_native/Cargo.lock` is committed for reproducible NIF builds. + +## [0.1.1] - 2026-06-15 + +### Added + +- **Blosc2 bit shuffle support** — `shuffle: :bit` option for `ExCodecs.encode(:blosc2, ...)`, + providing better compression on certain data patterns beyond the existing `:none` and `:byte` + shuffle modes (#20). + +### Fixed + +- **Livebook 02** — Fixed `Jason.encode!/1` wrapping for-comprehensions in `%{}` (#21). +- **Livebook 02** — Wrapped for-comprehensions in parentheses inside map values. +- **Livebook 03** — Removed `Kino.VegaLite.new()` calls (Livebook auto-renders VegaLite pipelines). +- **Livebook 03** — Fixed `Kino.Input.select` to use keyword lists (`atom: "label"`) instead + of maps. +- **Livebook 03** — Added `Kino.render()` wrapper for `Kino.Layout.grid()` widget. +- **Livebook 04** — Rewrote entirely: fixed broken markdown/code cell boundaries, moved + top-level `def` calls into `defmodule` blocks, fixed `Agent.get_and_update` state + corruption (was replacing struct with bare map), added `Kino`/`kino_vega_lite` deps. +- **Livebook 05** — Added `kino_vega_lite` dep for proper VegaLite rendering. + +### Changed + +- All livebooks now use **conditional Mix.install** setup cells that detect whether the + project is being run locally (from the repo) or published (from Hex.pm). Local + development uses `path:` deps with `rustler` and `force_build`; published livebooks + use `{:ex_codecs, "~> 0.1.1"}` with pre-compiled NIFs. +- README badges updated with CI, Hex.pm, docs, license, Elixir version, and Coveralls + coverage links. + +## [0.1.0] - 2025-06-09 + +### Added + +- Initial release of ExCodecs. +- Unified `encode/3` and `decode/3` API across all codecs. +- Codec registry with `available_codecs/0`, `supports?/1`, `codec_info/1`. +- Five compression codecs: Zstd, LZ4, Snappy, Bzip2, Blosc2. +- Blosc2 shuffle support (`:none`, `:byte`). +- Rust NIF implementation via `rustler_precompiled`. +- Precompiled binaries for macOS (ARM64, x86_64), Linux (glibc, musl, ARM64), Windows (x86_64). +- `ExCodecs.Compression` convenience module (`compress/2`, `decompress/2`). +- Structured error handling with `%ExCodecs.Error{}`. +- 154 tests (unit + property-based with StreamData). +- 90%+ test coverage. +- CI pipeline (test matrix, lint, coverage, docs). +- Release workflow for Hex.pm publishing with precompiled NIFs. +- Five livebooks: introduction, fundamentals, comparison, storage systems, Zarr workloads. +- Eleven guides covering all public modules and API details. +- Benchmarks via benchee. +- Credo and Dialyzer integration. \ No newline at end of file diff --git a/README.md b/README.md index e27e4e6..cab55bd 100644 --- a/README.md +++ b/README.md @@ -8,21 +8,28 @@ [![Elixir](https://img.shields.io/badge/Elixir-%7E%3E%201.17-purple.svg)](https://elixir-lang.org) [![Coverage Status](https://coveralls.io/repos/github/thanos/codecs/badge.svg?branch=main)](https://coveralls.io/github/thanos/codecs?branch=main) -An extensible BEAM-native codec framework for Elixir. +An extensible BEAM-native codec framework for Elixir with specialized category +APIs. Binary registry codecs use `ExCodecs.encode/3` / `decode/3`; +`ExCodecs.Spatial` handles point-cloud and Gaussian domain types. -ExCodecs provides a unified API for compression, decompression, hashing, checksums, -binary encodings, and future content-addressing codecs — all backed by Rust NIFs -for high throughput on the BEAM. +**Blosc2** produces C-Blosc2-compatible **chunks** only (not super-chunk / +B2ND / `.b2frame`). Standalone `:snappy` is separate from Blosc2 `cname:`. ## Design Philosophy ExCodecs is not a compression library. It is a codec framework. -Compression is merely the first codec category. The architecture supports future -expansion into hashing, checksums, binary encodings, content addressing, and -streaming -- without changing the public API. Every codec implements the -`ExCodecs.Codec` behaviour and registers with the `ExCodecs.CodecRegistry` at -startup, meaning new categories slot in without touching existing code. +Compression and spatial formats are categories in one framework. Each category +uses an API shaped for its data: + +- Binary→binary registry codecs implement `ExCodecs.Codec` and use + `ExCodecs.encode/3` / `decode/3`. +- Spatial struct↔format codecs use `ExCodecs.Spatial`. + +Discovery is category-specific: `ExCodecs.available_codecs/0` lists registered +binary codecs, while `ExCodecs.Spatial.available_formats/0` lists spatial +formats. This keeps one framework and error model without forcing unlike data +shapes through an overloaded function. The `encode`/`decode` naming is category-agnostic: for compression codecs, encoding is compressing and decoding is decompressing; for a future hash codec, @@ -35,7 +42,7 @@ Add `ex_codecs` to your list of dependencies in `mix.exs`: ```elixir def deps do [ - {:ex_codecs, "~> 0.1.1"} + {:ex_codecs, "~> 0.2.0"} ] end ``` @@ -48,61 +55,80 @@ mix deps.get && mix compile Precompiled NIF binaries are available for macOS (Intel and ARM64), Linux (x86_64 and ARM64, glibc and musl), and Windows (x86_64). They are downloaded -automatically from the [GitHub releases](https://github.com/ex-codecs/ex_codecs/releases) +automatically from the [GitHub releases](https://github.com/thanos/codecs/releases) when you run `mix deps.get`. If a precompiled artifact is not available for your target, ExCodecs falls back to compiling the Rust NIF from source (requires -Rust 1.85+). +Rust 1.94+). The native crate is **pure Rust** (no C toolchain / system +compression libraries). ## Quick Start ```elixir -# Compress data +# Registry codecs (primary API — always codec atom + binary) {:ok, compressed} = ExCodecs.encode(:zstd, "hello world") {:ok, original} = ExCodecs.decode(:zstd, compressed) original #=> "hello world" -# Compression with options {:ok, compressed} = ExCodecs.encode(:zstd, my_binary, level: 9) {:ok, compressed} = ExCodecs.encode(:blosc2, my_binary, cname: :zstd, clevel: 5, shuffle: :byte) -# Discover available codecs -ExCodecs.available_codecs() #=> [:blosc2, :bzip2, :lz4, :snappy, :zstd] +# Category alias for compression +{:ok, compressed} = ExCodecs.Compression.compress(:lz4, data) +{:ok, original} = ExCodecs.Compression.decompress(:lz4, compressed) + +# Shared discovery +ExCodecs.available_codecs() #=> [:blosc2, :bzip2, :gsplat, :lz4, :ply, :snappy, :spatial_binary, :zstd] +ExCodecs.available_codecs(:compression) #=> [:blosc2, :bzip2, :lz4, :snappy, :zstd] +ExCodecs.available_codecs(:spatial) #=> [:gsplat, :ply, :spatial_binary] ExCodecs.supports?(:zstd) #=> true ExCodecs.codec_info(:zstd) #=> {:ok, %ExCodecs.Codec{name: :zstd, category: :compression, ...}} -# Convenience aliases for compression -{:ok, compressed} = ExCodecs.Compression.compress(:lz4, data) -{:ok, original} = ExCodecs.Compression.decompress(:lz4, compressed) +# Spatial category API (structs ↔ formats) +alias ExCodecs.Spatial.{Point, PointCloud} +cloud = PointCloud.new([Point.new(0.0, 0.0, 0.0, color: {255, 0, 0})]) +{:ok, ply} = ExCodecs.Spatial.encode(cloud, format: :ply) +{:ok, cloud} = ExCodecs.Spatial.decode(ply, format: :ply) +ExCodecs.Spatial.stream_decode(ply, format: :ply) |> Enum.to_list() +ExCodecs.Spatial.available_formats() #=> [:ply, :spatial_binary, :gsplat] +ExCodecs.codec_info(:ply) #=> {:ok, %ExCodecs.Codec{category: :spatial, interface: :spatial, ...}} ``` ## API Overview -### `encode/3` +### Binary registry API ```elixir {:ok, encoded} = ExCodecs.encode(:zstd, binary, level: 3) +{:ok, decoded} = ExCodecs.decode(:zstd, compressed) ``` -Encodes (compresses) data with the given codec and options. Returns -`{:ok, binary}` on success or `{:error, %ExCodecs.Error{}}` on failure. +The binary registry API is always **codec atom + binary**. -### `decode/3` +### Category APIs -```elixir -{:ok, decoded} = ExCodecs.decode(:zstd, compressed) -``` +- `ExCodecs.Compression.compress/3` — compression terminology over the binary + registry API. +- `ExCodecs.Spatial.encode/2` / `decode/2` — point clouds and Gaussians + (struct↔format). + +These are specialized entry points in the same framework, sharing conventions +such as tagged results and `%ExCodecs.Error{}`. + +### `encode/3` / `decode/3` -Decodes (decompresses) data with the given codec. Returns `{:ok, binary}` on -success or `{:error, %ExCodecs.Error{}}` on failure. +First argument is always a **codec atom**. Returns `{:ok, binary}` or +`{:error, %ExCodecs.Error{}}`. ### `available_codecs/0` ```elixir -ExCodecs.available_codecs() #=> [:blosc2, :bzip2, :lz4, :snappy, :zstd] +ExCodecs.available_codecs() # all available catalog entries +ExCodecs.available_codecs(:spatial) #=> [:gsplat, :ply, :spatial_binary] ``` Returns a sorted list of codec atoms that are both registered and have a -loaded native implementation. +loaded implementation. Use `available_codecs/1` to filter the shared catalog +by category. ### `supports?/1` @@ -119,9 +145,9 @@ Returns `true` only if the codec is registered **and** its native NIF is loaded. {:ok, info} = ExCodecs.codec_info(:zstd) info.category #=> :compression info.native? #=> true -info.streaming? #=> true +info.streaming? #=> false # block API only today info.configurable? #=> true -info.version #=> approximate (e.g., "1.5.x") +info.version #=> backend version string ``` Returns a structured `%ExCodecs.Codec{}` struct with metadata, or @@ -131,37 +157,57 @@ Returns a structured `%ExCodecs.Codec{}` struct with metadata, or | Codec | Category | Configurable | Streaming | Options | |-----------|-------------|--------------|-----------|------------------------------------------------| -| `:zstd` | compression | Yes | Yes | `level` (1-22, default 3) | -| `:lz4` | compression | No | No | -- | -| `:snappy` | compression | No | No | -- | -| `:bzip2` | compression | Yes | No | `block_size` (1-9, default 9) | -| `:blosc2` | compression | Yes | Yes | `cname`, `clevel` (0-9, default 5), `shuffle` (`:none` / `:byte` / `:bit`, default `:byte`), `typesize` (default 8) | +| `:zstd` | compression | Yes | No | `level` 1-22 (pure-Rust `structured-zstd`); `max_output_size` (default 256 MiB) | +| `:lz4` | compression | No | No | size-prepended `lz4_flex`; `max_output_size` | +| `:snappy` | compression | No | No | `max_output_size` | +| `:bzip2` | compression | Yes | No | `block_size` 1-9; `max_output_size` | +| `:blosc2` | compression | Yes | No | C-Blosc2 **chunk** only. `cname` / `clevel` / `shuffle` / `typesize`; `max_output_size` | + +### Spatial formats + +| Format | Types | Notes | +|-------------------|-------------------------------|----------------------------------------------------| +| `:ply` | PointCloud / GaussianCloud | ASCII or binary PLY; Gaussian PLY properties | +| `:spatial_binary` | PointCloud | Compact little-endian `EXCP` container | +| `:gsplat` | GaussianCloud | Compact little-endian `GSPL` container | + +See [Understanding Spatial Codecs](guides/understanding_spatial_codecs.md) and +the frozen wire layouts in [docs/spatial_formats.md](docs/spatial_formats.md). + +### Known limitations + +- **Zstd** uses pure-Rust `structured-zstd`. Levels 1–22 work, but compressed + bytes/ratios are not guaranteed identical to C libzstd. +- **Decompression** defaults to a **256 MiB** `max_output_size`. Raise it only + for trusted inputs; do not decompress untrusted payloads without a tight limit. +- Spatial `stream_*` helpers **materialize** full payloads today. ## Architecture ExCodecs is layered as follows: -1. **Public API** (`ExCodecs`) -- `encode/3`, `decode/3`, `available_codecs/0`, - `supports?/1`, `codec_info/1`. All consumers interact with this module. +1. **Category APIs** — `ExCodecs` provides the binary registry API; + `ExCodecs.Compression` adds compression terminology; `ExCodecs.Spatial` + handles spatial domain types and formats. -2. **Codec Behaviour** (`ExCodecs.Codec`) -- A behaviour requiring `encode/2` - and `decode/2` callbacks. Each codec module implements this behaviour and - optionally exports `__codec_info__/0` for registry metadata. +2. **Codec Behaviour** (`ExCodecs.Codec`) — `encode/2` / `decode/2` on binaries; + optional `__codec_info__/0` for registry metadata. -3. **Codec Registry** (`ExCodecs.CodecRegistry`) -- An ETS-backed registry - populated at application startup. It maps codec atoms to their implementing - modules and metadata. Lookups are O(1). +3. **Shared codec catalog** (`ExCodecs.CodecRegistry`) — ETS map of codec atoms + to modules, categories, interface shapes, and metadata, populated at startup. -4. **Native NIFs** (`ExCodecs.Native`) -- Rustler NIFs providing the actual - compression and decompression. Precompiled via `rustler_precompiled` for - cross-platform distribution; falls back to local compilation. +4. **Native NIFs** (`ExCodecs.Native`) — pure-Rust compression via + `rustler_precompiled` (or local compile). -5. **Category Modules** (`ExCodecs.Compression`) -- Convenience modules that - delegate to the public API with category-specific naming (e.g., - `compress`/`decompress`). +5. **Category discovery** — `available_codecs/0` lists the whole catalog; + `available_codecs/1` filters it, and + `ExCodecs.Spatial.available_formats/0` provides the spatial category's + preferred built-in order. -To add a new codec: implement `ExCodecs.Codec`, add a native NIF function, -register the codec in `ExCodecs.Application`, and the rest follows automatically. +To add a binary codec: implement `ExCodecs.Codec`, wire a NIF if needed, and +register it with `interface: :binary`. Spatial implementations live under +`ExCodecs.Spatial` and register with `category: :spatial` and +`interface: :spatial`. ## Error Handling @@ -177,6 +223,9 @@ Error reasons are atoms: | `:compression_failed` | The underlying compression operation failed | | `:decompression_failed`| The underlying decompression operation failed | | `:nif_not_loaded` | The NIF library could not be loaded | +| `:io_error` | File read/write failure | +| `:truncated_input` | Incomplete binary | +| `:output_limit_exceeded` | Decompress would exceed `max_output_size` | ```elixir {:error, error} = ExCodecs.encode(:unknown, "data") @@ -193,9 +242,6 @@ ExCodecs includes benchmarking utilities via [benchee](https://github.com/benche ```sh # Run all compression benchmarks mix benchmarks - -# Run a specific benchmark file -mix bench compression ``` Benchmarks are defined in `bench/` and run in the `:bench` environment. Results @@ -234,7 +280,7 @@ mix docs - Elixir 1.17+ - Erlang/OTP 26+ -- Rust 1.85+ (only required if precompiled NIFs are unavailable for your platform) +- Rust 1.94+ (only required if precompiled NIFs are unavailable for your platform) ## License diff --git a/docs/architecture.md b/docs/architecture.md index d1789e1..7118de7 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -20,8 +20,8 @@ A production-quality, extensible BEAM-native codec framework for Elixir. ## Design Philosophy -ExCodecs is not a compression library. It is a codec framework where -compression happens to be the first implemented category. +ExCodecs is not only a compression library. It is a codec framework with +compression and spatial categories. The central insight is that many binary transformations -- compression, hashing, checksums, binary encodings, content addressing -- share the same @@ -37,10 +37,11 @@ Two terminology decisions flow from this insight: tag. Base64 encodes binary into ASCII. The encode/decode terminology unifies all codec categories under one mental model. -2. **Codec framework, not compression library.** The architecture must support - adding, discovering, and querying codecs at runtime without changing the - public API. A new codec category should require adding a namespace module - and implementing two callbacks -- nothing more. +2. **Codec framework, not compression library.** Categories belong to one + framework but may expose APIs suited to their data shape. Binary→binary + codecs share the registry behaviour; domain codecs such as spatial formats + use a category-specific contract. All categories retain the same tagged + result and structured-error conventions. The result is an API where `ExCodecs.encode(:zstd, data)` and `ExCodecs.encode(:sha256, data)` look and feel the same, even though one @@ -51,61 +52,61 @@ reversibly compresses data and the other irreversibly hashes it. ## Architecture Overview ``` -+---------------------------------------------------------------+ -| Public API | -| ExCodecs.encode/3 ExCodecs.decode/3 | -| ExCodecs.available_codecs/0 ExCodecs.supports?/1 | -| ExCodecs.codec_info/1 | -+----------------------------+----------------------------------+ - | - +--------v--------+ - | CodecRegistry | - | (ETS-backed) | - +--------+--------+ - | - +--------------+---------------+ - | | | - +---------v----+ +------v-----+ +-----v------+ - | :compression | | :hashing | | :checksum | (future) - | namespace | | (future) | | (future) | - +---------+----+ +------------+ +------------+ - | - +---------+----------+-----------+----------+ - | | | | | - +---+ +---+ +-------+ +-----+ +--------+ - |Zstd| |LZ4| |Snappy| |Bzip2| |Blosc2 | - +---+ +---+ +-------+ +-----+ +--------+ - | | | | | - +---+---+--------+----------+----------+ - | - +-----v------+ - | Native NIF | (Rustler, DirtyCpu scheduling) - +------------+ - | - +-----v-----+ - | Rust impl | (zstd, lz4_flex, snap, bzip2, pure Rust blosc2) - +-----------+ -``` - -The architecture is layered in four tiers: ++------------------------------------------------------------------+ +| Public discovery | +| available_codecs/0,1 supports?/1 codec_info/1 | ++-------------------------------+----------------------------------+ + | + +--------v---------+ + | Shared catalog | + | CodecRegistry/ETS| + +---+----------+---+ + | | + +------------v--+ +--v----------------+ + | :compression | | :spatial | + | interface: | | interface: | + | :binary | | :spatial | + +-------+--------+ +---------+----------+ + | | + ExCodecs.encode/3, decode/3 ExCodecs.Spatial.encode/2, + | decode/2, stream helpers + +--------------+------+ +---------+---------+ + | Zstd/LZ4/Snappy/... | | PLY / EXCP / GSPL | + +--------------+------+ +-------------------+ + | + +--------v---------+ + | Rustler NIFs | + | pure-Rust codecs | + +------------------+ +``` + +Discovery is unified; operation dispatch is category-safe. The catalog's +`interface` metadata prevents top-level binary encode/decode from accidentally +calling a struct↔format codec. `ExCodecs.Spatial` resolves its implementations +from the same catalog. + +The implementation is layered in four tiers: 1. **Public API** -- The user-facing `ExCodecs` module providing `encode`, `decode`, `available_codecs`, `supports?`, and `codec_info`. -2. **Registry** -- The `CodecRegistry` module backed by an ETS table that maps - codec atoms to module references, category atoms, and metadata. +2. **Shared catalog** -- The `CodecRegistry` module backed by an ETS table that + maps codec atoms to modules, categories, interface shapes, and metadata. -3. **Codec modules** -- Individual modules implementing `ExCodecs.Codec`, one - per codec algorithm. Each module validates options and delegates to the NIF. +3. **Codec modules** -- Binary modules implement `ExCodecs.Codec`; spatial + modules expose the category's struct↔format contract. -4. **Native layer** -- The Rustler NIF (`ExCodecs.Native`) providing the actual - algorithmic implementations compiled from Rust. +4. **Native layer** -- Compression modules delegate to the Rustler NIF + (`ExCodecs.Native`). Spatial codecs are Elixir implementations in v0.2.0. --- ## Public API -The public API surface is intentionally small: five functions. +The binary registry API is intentionally small (encode/decode plus discovery). +Spatial uses `ExCodecs.Spatial.encode/2` / `decode/2`; top-level +`stream_encode/2` and `stream_decode/2` are convenience delegates for spatial +data. Top-level `encode/3` / `decode/3` are not overloaded for spatial structs. ```elixir # Encoding and decoding @@ -113,7 +114,9 @@ ExCodecs.encode(:zstd, data, level: 3) #=> {:ok, binary} | {:error, Error. ExCodecs.decode(:zstd, compressed) #=> {:ok, binary} | {:error, Error.t()} # Discovery -ExCodecs.available_codecs() #=> [:bzip2, :blosc2, :lz4, :snappy, :zstd] +ExCodecs.available_codecs() #=> [..., :gsplat, :ply, :spatial_binary, ...] +ExCodecs.available_codecs(:compression) #=> [:blosc2, :bzip2, :lz4, :snappy, :zstd] +ExCodecs.available_codecs(:spatial) #=> [:gsplat, :ply, :spatial_binary] ExCodecs.supports?(:zstd) #=> true | false ExCodecs.codec_info(:zstd) #=> {:ok, %Codec{}} | {:error, :unsupported_codec} ``` @@ -240,11 +243,12 @@ def __codec_info__ do %ExCodecs.Codec{ name: :zstd, category: :compression, + interface: :binary, module: __MODULE__, native?: true, - streaming?: true, + streaming?: false, configurable?: true, - version: "1.5.6" + version: "structured-zstd-0.0.48" } end ``` @@ -255,6 +259,7 @@ The `%ExCodecs.Codec{}` struct fields: |-----------------|-------------------|-------------------------------------------------| | `name` | `atom()` | The codec's registry key | | `category` | `atom()` | Category (`:compression`, `:hashing`, etc.) | +| `interface` | `:binary \| :spatial` | Operation API shape | | `module` | `module() \| nil` | The implementing module, or nil if unavailable | | `native?` | `boolean()` | Whether the codec uses a NIF | | `streaming?` | `boolean()` | Whether the codec supports streaming | @@ -268,7 +273,7 @@ that add complexity without benefit. Instead, the registry probes for the function with `function_exported?/3`: ```elixir -defp build_codec_info(name, module, category) do +defp build_codec_info(name, module, category, interface, metadata) do info = if function_exported?(module, :__codec_info__, 0) do module.__codec_info__() @@ -279,11 +284,12 @@ defp build_codec_info(name, module, category) do %ExCodecs.Codec{ name: name, category: category, + interface: interface, module: module, - native: Map.get(info, :native?, true), - streaming?: Map.get(info, :streaming?, false), - configurable?: Map.get(info, :configurable?, false), - version: Map.get(info, :version) + native?: Keyword.get(metadata, :native?, info.native?), + streaming?: Keyword.get(metadata, :streaming?, info.streaming?), + configurable?: Keyword.get(metadata, :configurable?, info.configurable?), + version: Keyword.get(metadata, :version, info.version) } end ``` @@ -329,13 +335,16 @@ The Elixir side is defined in `ExCodecs.Native`: ```elixir defmodule ExCodecs.Native do - use Rustler, + use RustlerPrecompiled, otp_app: :ex_codecs, crate: :ex_codecs_native, - mode: :release + # ... version, base_url, targets ... end ``` +The native crate is pure Rust (structured-zstd, lz4_flex, snap, flate2 rust backend, +libbz2-rs) — no C compression libraries. + Each NIF function has a fallback that returns `:erlang.nif_error(:nif_not_loaded)`: ```elixir @@ -443,15 +452,15 @@ strip = true ## Runtime Codec Discovery -The `ExCodecs.CodecRegistry` is an ETS-backed Agent that maps codec names -to their implementations and metadata. +The `ExCodecs.CodecRegistry` is an ETS-backed Agent that serves as the shared +catalog for binary and spatial codec names, implementations, and metadata. ### Why ETS? ETS offers O(1) lookups for `set`-type tables. Codec resolution happens on -every `encode`/`decode` call, so the registry must be fast. A GenServer with -a map would serialize all codec lookups through a single process; ETS allows -concurrent reads without bottlenecks. +binary and spatial encode/decode calls, so the catalog must be fast. A +GenServer with a map would serialize all lookups through a single process; ETS +allows concurrent reads without bottlenecks. ### Registry data model @@ -466,12 +475,13 @@ Example entries: ```elixir {:zstd, {ExCodecs.Compression.Zstd, :compression, %Codec{name: :zstd, ...}}} {:lz4, {ExCodecs.Compression.Lz4, :compression, %Codec{name: :lz4, ...}}} +{:ply, {ExCodecs.Spatial.Codec.PLY, :spatial, %Codec{name: :ply, interface: :spatial, ...}}} {:sha256, {nil, :hashing, %Codec{name: :sha256, module: nil, ...}}} # unavailable ``` -When a codec's NIF fails to load, the module field is `nil` and the codec is -registered as unavailable. Calls to `encode`/`decode` with an unavailable -codec return `{:error, %ExCodecs.Error{reason: :codec_unavailable}}`. +When an implementation cannot load, the module field is `nil` and the codec is +registered as unavailable. Calls through its category API return +`{:error, %ExCodecs.Error{reason: :codec_unavailable}}`. ### Discovery flow @@ -487,7 +497,11 @@ codec return `{:error, %ExCodecs.Error{reason: :codec_unavailable}}`. +---> {:ok, {ExCodecs.Compression.Zstd, :compression, info}} | | | v - | info.module != nil? --> yes --> module.encode(data, opts) + | info.interface == :binary and module != nil? + | | + | yes --> module.encode(data, opts) + | + +---> spatial interface --> return category-API guidance | +---> {:error, :unsupported_codec} | @@ -499,17 +513,23 @@ codec return `{:error, %ExCodecs.Error{reason: :codec_unavailable}}`. ```elixir # List all codecs with available implementations -ExCodecs.available_codecs() #=> [:bzip2, :blosc2, :lz4, :snappy, :zstd] +ExCodecs.available_codecs() #=> [:blosc2, ..., :gsplat, :ply, ..., :zstd] + +# Filter available names by category +ExCodecs.available_codecs(:spatial) #=> [:gsplat, :ply, :spatial_binary] # List all registered codecs (including unavailable) -ExCodecs.CodecRegistry.all_codecs() #=> [:blosc2, :bzip2, :lz4, :sha256, :snappy, :zstd] +ExCodecs.CodecRegistry.all_codecs() #=> [:blosc2, ..., :ply, :sha256, ..., :zstd] # Check if a specific codec is available ExCodecs.supports?(:zstd) #=> true # Get detailed metadata ExCodecs.codec_info(:zstd) -#=> {:ok, %Codec{name: :zstd, category: :compression, native?: true, ...}} +#=> {:ok, %Codec{name: :zstd, category: :compression, interface: :binary, ...}} + +ExCodecs.codec_info(:ply) +#=> {:ok, %Codec{name: :ply, category: :spatial, interface: :spatial, ...}} # Filter by category ExCodecs.Compression.available_codecs() @@ -705,7 +725,25 @@ predictable and navigable. --- -## Future Categories +## Codec Categories + +### Spatial (implemented in 0.2.0) + +Spatial codecs map structured geometric types to interchange formats. They are +pure Elixir and use the specialized `ExCodecs.Spatial` API rather than forcing +structs through the binary-only `ExCodecs.Codec` callbacks. They are registered +in the shared catalog with `category: :spatial` and `interface: :spatial`. + +`ExCodecs.available_codecs/0` lists all available entries, +`ExCodecs.available_codecs(:spatial)` filters the shared catalog, and +`ExCodecs.Spatial.available_formats/0` presents spatial formats in the +category's preferred built-in order. + +| Format | Module | +|-------------------|-------------------------------------| +| `:ply` | `ExCodecs.Spatial.Codec.PLY` | +| `:spatial_binary` | `ExCodecs.Spatial.Codec.Binary` | +| `:gsplat` | `ExCodecs.Spatial.Codec.Gsplat` | ### Hashing @@ -790,7 +828,8 @@ codec, then assembles the result according to the CID specification. --- -The architecture described here is built to accommodate all of these categories -without API changes. Every new codec, regardless of category, implements the -same two-callback behaviour, registers in the same ETS table, and is queried -through the same five public functions. \ No newline at end of file +The architecture described here accommodates categories through specialized +entry points under one framework. Binary-shaped categories can reuse +`ExCodecs.Codec` and the ETS registry; domain-shaped categories may define +their own behaviour and discovery while preserving common result, error, and +documentation conventions. \ No newline at end of file diff --git a/docs/spatial_formats.md b/docs/spatial_formats.md new file mode 100644 index 0000000..4fc83f5 --- /dev/null +++ b/docs/spatial_formats.md @@ -0,0 +1,116 @@ +# Spatial wire formats (v0.2.0 freeze) + +This document freezes the on-wire layouts used by `ExCodecs.Spatial` in **v0.2.0**. +A future Rust backend (v0.2.1) must produce **byte-compatible** output for these +formats. There is **no CRC / checksum** in v1; integrity is the caller's responsibility. + +Attribute keys on points are **strings** after construction / decode. Prefer string +keys when building clouds for cross-backend compatibility. + +## One framework, specialized category APIs + +| Category | Entry point | Discovery | +|----------|-------------|-----------| +| Registry binary codecs (`:zstd`, …) | `ExCodecs.encode/3`, `decode/3` | `available_codecs(:compression)` | +| Spatial domain codecs | `ExCodecs.Spatial.encode/2`, `decode/2` | `available_codecs(:spatial)` or `ExCodecs.Spatial.available_formats/0` | + +Both categories belong to the ExCodecs framework and share its tagged-result +and error conventions. Their entry points differ because registry codecs map +binary↔binary, while spatial codecs map domain structs↔formats. + +Spatial formats (`:ply`, `:spatial_binary`, `:gsplat`) are entries in the same +ETS-backed catalog as compression codecs. Their metadata has +`category: :spatial` and `interface: :spatial`; that interface marker keeps +operation dispatch on `ExCodecs.Spatial` rather than the binary API. + +## PLY (`format: :ply`) + +Standard PLY interchange (ASCII or binary little/big endian). Gaussian PLY uses +common property names (`f_dc_*`, `opacity`, `scale_*`, `rot_*`, `f_rest_*`). + +Encode options (after Spatial selects `:ply`): + +- `:ply_format` or `:format` — `:ascii` (default), `:binary` / `:binary_le`, `:binary_be` +- `:comments` — header comments +- `:as` — decode as `:auto`, `:point_cloud`, or `:gaussian_cloud` + +### Schema promotion + +If **any** point in a cloud has color / alpha / normals, the schema includes those +properties for **all** points. Missing values are defaulted (RGB `0`, alpha `255`, +normal `{0,0,0}`). Mixed optional fields therefore round-trip with defaults filled in. + +### Streaming + +`stream_decode` / `stream_encode` currently **materialize** the full payload (or +enumerable) then enumerate. Prefer explicit `source: :file` or `source: :binary` +when the argument is ambiguous. + +`:auto` treats a binary as a path only when it looks path-like (under 4 KiB, no +`ply`/`EXCP`/`GSPL` magic, and contains `/` or `\` or ends with `.ply`/`.excp`/ +`.gspl`/`.bin`) **and** `File.regular?/1` is true. Edge cases: + +- A real file with no separator/extension is **not** auto-opened. +- A short binary containing `/` that happens to be a regular file path **is** opened. + +## EXCP — `:spatial_binary` (version 1) + +Little-endian compact point clouds. + +``` +Offset Size Field +0 4 magic "EXCP" +4 2 version u16 LE (= 1) +6 2 flags u16 LE +8 8 count u64 LE +16 … records +``` + +**Flags** + +| Bit | Meaning | +|-----|---------| +| 0 | color (RGB u8 × 3) | +| 1 | alpha (RGBA; implies color bytes include alpha) | +| 2 | normals (f32 × 3) | + +**Record order:** `x,y,z` as `f32` LE, then optional color bytes, then optional +normals. Flags are global for the file; points missing a promoted field are +zero-filled (alpha default 255). + +**Not stored:** generic attributes, bounds, transform, metadata. Trailing bytes +after `count` records are ignored. Truncation yields `:invalid_data` / +`:truncated_input` as implemented. + +## GSPL — `:gsplat` (version 1) + +Little-endian compact Gaussian clouds. + +``` +Offset Size Field +0 4 magic "GSPL" +4 2 version u16 LE (= 1) +6 2 flags u16 LE (bit0 set when SH rest present) +8 8 count u64 LE +16 2 sh_rest u16 LE (floats of SH rest per Gaussian; 0 if none) +18 … records +``` + +**Record (always):** 14 × `f32` LE — + +1. position XYZ (3) +2. DC color RGB (3) — maps to `Gaussian.color` +3. opacity (1) +4. scale XYZ (3) +5. rotation quaternion WXYZ (4) + +Then `sh_rest` × `f32` LE (shared length for all Gaussians; shorter lists padded +with zeros on encode). + +**Not stored:** per-Gaussian metadata maps, cloud metadata. Flags on decode are +currently informational; `sh_rest` count in the header is authoritative. + +## Integrity + +v1 formats have **no checksum**. For integrity, wrap payloads with a registry +codec (e.g. compress after encode) or store an external hash. diff --git a/guides/choosing_compression_codec.md b/guides/choosing_compression_codec.md index ea02207..f5cc924 100644 --- a/guides/choosing_compression_codec.md +++ b/guides/choosing_compression_codec.md @@ -2,19 +2,59 @@ This guide helps you select the right compression codec for your use case. It includes a comparison table, decision criteria, and worked examples. +> ### Is the data spatial—or spatially representable? +> +> Compression codecs operate on arbitrary binaries. If records represent +> coordinates, points, normals, colors, oriented particles, or Gaussian splats, +> consider converting them to `ExCodecs.Spatial` types first—even when the +> source is currently a flat array, CSV rows, maps, or database records. +> +> Use: +> +> - `ExCodecs.Spatial.PointCloud` for XYZ, XYZRGB(A), normals, and scalar +> per-point attributes. +> - `ExCodecs.Spatial.GaussianCloud` for position, scale, rotation, opacity, +> color, and spherical harmonics. +> - `:ply` for interoperable spatial exchange. +> - `:spatial_binary` (EXCP) for compact point clouds. +> - `:gsplat` (GSPL) for compact Gaussian clouds. +> +> Spatial encoding and compression are complementary. Encode the structured +> data first, then optionally compress the resulting binary: +> +> ```elixir +> alias ExCodecs.Spatial.{Point, PointCloud} +> +> cloud = +> rows +> |> Enum.map(fn %{x: x, y: y, z: z} -> Point.new(x, y, z) end) +> |> PointCloud.new() +> +> {:ok, spatial_binary} = +> ExCodecs.Spatial.encode(cloud, format: :spatial_binary) +> +> {:ok, compressed} = ExCodecs.encode(:zstd, spatial_binary, level: 3) +> ``` +> +> If the numbers have no geometric meaning and only need compact storage, +> Blosc2 may be a better fit. See +> [Understanding Spatial Codecs](understanding_spatial_codecs.md) and +> [Spatial Wire Formats](../docs/spatial_formats.md). + ## Quick Comparison | Codec | Compression Ratio | Compression Speed | Decompression Speed | Memory Usage | Configurable | Streaming | Best For | |----------|-------------------|-------------------|---------------------|--------------|--------------|-----------|-----------------------------------| -| LZ4 | Low | Very Fast | Very Fast | Very Low | Level 1-16 | No | Real-time, latency-sensitive | +| LZ4 | Low | Very Fast | Very Fast | Very Low | No | No | Real-time, latency-sensitive | | Snappy | Low | Very Fast | Very Fast | Very Low | No | No | Short-lived data, RPC payloads | -| Zstd | High | Fast | Very Fast | Moderate | Level 1-22 | Yes | General purpose, storage, network | +| Zstd | High | Fast | Very Fast | Moderate | Levels 1-22 | No | General purpose, storage, network | | Bzip2 | Very High | Slow | Slow | Moderate | Block 1-9 | No | Archival, offline processing | -| Blosc2 | High* | Fast | Very Fast** | Configurable | Extensive | Yes | Numerical arrays, typed data | +| Blosc2 | High* | Fast | Very Fast | Moderate | Codec/level/shuffle/typesize | No | Numerical arrays, typed data | \* Blosc2 ratio depends heavily on the internal codec (`cname`) and shuffle settings. With byte shuffle on typed data, ratios can exceed Zstd. -\** Blosc2 decompression is fast because it can skip the decompression of unused blocks. +“Streaming” here means an incremental compression API. All current compression +codecs operate on complete input buffers. ## Decision Framework @@ -22,8 +62,11 @@ Answer the following questions to narrow your choice: ### 1. What is your data type? +- **Coordinates, point samples, particles, normals, or Gaussian splats** -- + model them with `ExCodecs.Spatial`, even if they currently arrive as arrays, + rows, maps, or CSV. Optionally compress the encoded PLY/EXCP/GSPL binary. - **Text, JSON, logs, general-purpose binary** -- Zstd is the best default. Use level 3 for a balance of speed and ratio. -- **Typed numerical arrays (floats, integers, matrices)** -- Blosc2 with appropriate `typesize` and `shuffle` settings. +- **Typed numerical arrays without spatial semantics (floats, integers, matrices)** -- Blosc2 with appropriate `typesize` and `shuffle` settings. - **Short-lived messages, RPC payloads** -- Snappy or LZ4 for minimal latency. - **Archival storage** -- Bzip2 for maximum ratio, or Zstd at level 19-22 for high ratio with better decompression speed. @@ -43,7 +86,7 @@ Answer the following questions to narrow your choice: - **Very constrained (embedded, large concurrent load)** -- LZ4 or Snappy. - **Moderate** -- Zstd at levels 1-14. -- **Available** -- Zstd at levels 15-22, Bzip2 at block sizes 7-9, Blosc2 multi-threaded. +- **Available** -- Zstd at levels 15-22 or Bzip2 at block sizes 7-9. ### 5. Is data read once or many times? @@ -57,9 +100,9 @@ Answer the following questions to narrow your choice: ```elixir {:ok, compressed} = ExCodecs.encode(:lz4, data) -{:ok, compressed} = ExCodecs.encode(:lz4, data, level: 4) ``` +- ExCodecs exposes one fixed, fast LZ4 profile; it does not accept `:level`. - Low-latency message queues - Real-time data pipelines where throughput matters more than size - Temporary data that will be decompressed quickly @@ -87,7 +130,8 @@ Answer the following questions to narrow your choice: - Databases, file storage, and network transmission - Workloads where decompression speed matters (Zstd decompresses fast regardless of compression level) - When you need a configurable tradeoff (22 levels from fast to maximum ratio) -- Dictionary compression for small, repetitive payloads + +ExCodecs does not currently expose Zstd dictionary compression. ### When to Use Bzip2 @@ -109,9 +153,11 @@ Answer the following questions to narrow your choice: - Numerical arrays (float64, int32, etc.) - Scientific data, time series, matrix storage - Situations where shuffle filters provide a significant ratio improvement -- Multi-threaded compression/decompression of large buffers - When you need fine-grained control over the compression pipeline +Each Blosc2 NIF call is single-threaded (`nthreads: 1`). Parallelize independent +buffers with BEAM processes. + ## Worked Examples ### Example 1: API Response Cache @@ -134,13 +180,14 @@ Rationale: Zstd decompresses quickly regardless of compression level, so invest Messages arrive at high volume and must be forwarded with minimal latency. -**Choice: LZ4 at level 1** +**Choice: LZ4** ```elixir -{:ok, compressed} = ExCodecs.encode(:lz4, message, level: 1) +{:ok, compressed} = ExCodecs.encode(:lz4, message) ``` -Rationale: Latency is the priority. LZ4 at level 1 provides compression at over 500 MB/s, adding negligible overhead to the pipeline. +Rationale: Latency is the priority. ExCodecs' fixed fast LZ4 profile adds +minimal overhead to the pipeline. ### Example 3: Scientific Data Archive @@ -153,8 +200,7 @@ A research pipeline archives float64 measurement arrays to cold storage. cname: :zstd, clevel: 9, shuffle: :byte, - typesize: 8, - numthreads: 4 + typesize: 8 ) ``` @@ -191,10 +237,11 @@ Rationale: Protobuf already removes much redundancy. Snappy adds minimal overhea | Use Case | Recommended Codec | Configuration | |------------------------------|-------------------|-------------------------------------| | General-purpose default | Zstd | `level: 3` | -| Real-time / low-latency | LZ4 | `level: 1` | +| Real-time / low-latency | LZ4 | No codec-specific options | | Fastest with no config | Snappy | (none) | | Maximum ratio / archival | Bzip2 or Zstd | `block_size: 9` or `level: 19-22` | | Numerical arrays | Blosc2 | `cname: :zstd, shuffle: :byte` | -| Small repetitive payloads | Zstd | `level: 3` with dictionary | -| In-memory cache | LZ4 or Snappy | `level: 1` or (none) | -| Already slightly compressed | Snappy or LZ4 | Lowest level to avoid wasted CPU | \ No newline at end of file +| Spatially meaningful records | Spatial + optional compression | PLY/EXCP/GSPL, then optionally Zstd | +| Small repetitive payloads | Zstd | `level: 3` (dictionary API unavailable) | +| In-memory cache | LZ4 or Snappy | No codec-specific options | +| Already slightly compressed | Snappy or LZ4 | No codec-specific options | \ No newline at end of file diff --git a/guides/codec_fundamentals.md b/guides/codec_fundamentals.md index 8bf7af7..0e85b4b 100644 --- a/guides/codec_fundamentals.md +++ b/guides/codec_fundamentals.md @@ -26,10 +26,14 @@ Every codec in ExCodecs implements two functions: Key properties of this design: -- **Binary in, binary out.** Codecs operate on raw binaries. Structured data must be serialized before encoding. +- **Binary in, binary out** for **registry** codecs. Structured data is either + serialized first, or handled by a **category module** (e.g. spatial structs + via `ExCodecs.Spatial`). - **Optionally configurable.** The `opts` keyword list allows codec-specific tuning (compression level, block size, shuffle mode) while keeping the call signature uniform. - **Explicit error handling.** Every operation returns `{:ok, result}` or `{:error, %ExCodecs.Error{}}`. There are no exceptions for normal failure modes. - **Composable.** Because the input and output share the same type, codecs can be chained: encode with Zstd, then encode the result with Blosc2 if desired. +- **Category modules** provide domain naming (`Compression.compress/3`) or + non-binary shapes (`Spatial.encode/2`) without overloading the registry API. ### Compressing Data @@ -96,9 +100,9 @@ def __codec_info__ do category: :compression, module: __MODULE__, native?: true, - streaming?: true, + streaming?: false, configurable?: true, - version: "1.5.6" + version: "structured-zstd-0.0.48" } end ``` @@ -111,7 +115,7 @@ The `%ExCodecs.Codec{}` struct contains: | `category` | `atom()` | Category (e.g., `:compression`) | | `module` | `module() \| nil` | Implementing module, or `nil` if unavailable | | `native?` | `boolean()` | Whether a native NIF implementation exists | -| `streaming?` | `boolean()` | Whether the codec supports streaming operation | +| `streaming?` | `boolean()` | Whether an incremental encode/decode API is exposed | | `configurable?` | `boolean()` | Whether the codec accepts configuration options | | `version` | `String.t() \| nil` | Library version string | diff --git a/guides/runtime_codec_discovery.md b/guides/runtime_codec_discovery.md index 64d53fb..ae15256 100644 --- a/guides/runtime_codec_discovery.md +++ b/guides/runtime_codec_discovery.md @@ -1,6 +1,9 @@ # Runtime Codec Discovery -ExCodecs uses a runtime registry to discover, validate, and query available codecs. This guide explains how the registry works, how it enables graceful degradation, and how to extend it with custom codecs. +ExCodecs uses a shared runtime catalog to discover, validate, and query binary +and spatial codecs. This guide explains how the ETS-backed registry works, how +it enables graceful degradation, and how to extend it with custom binary +codecs. ## The Codec Registry @@ -73,7 +76,10 @@ For an unavailable codec: The distinction between "known but unavailable" and "unknown" is important: -- **Known but unavailable**: The codec is registered in the table with `module: nil`. This means the native NIF could not be loaded (e.g., unsupported platform, missing native library). `supports?/1` returns `false`, but `codec_info/1` still returns the metadata. +- **Known but unavailable**: The codec is registered in the table with + `module: nil`. Its implementation could not be loaded (for compression this + commonly means the native NIF is unavailable). `supports?/1` returns + `false`, but `codec_info/1` still returns metadata. - **Unknown**: The codec name is not in the table at all. `lookup/1` returns `{:error, :unsupported_codec}`. @@ -81,11 +87,17 @@ The distinction between "known but unavailable" and "unknown" is important: ### Available Codecs -List all codecs that are loaded and functional: +List all catalog entries that are loaded and functional: ```elixir ExCodecs.available_codecs() +# => [:blosc2, :bzip2, :gsplat, :lz4, :ply, :snappy, :spatial_binary, :zstd] + +ExCodecs.available_codecs(:compression) # => [:blosc2, :bzip2, :lz4, :snappy, :zstd] + +ExCodecs.available_codecs(:spatial) +# => [:gsplat, :ply, :spatial_binary] ``` This filters out codecs where the module is `nil` (unavailable). Only codecs you can actually use are included. @@ -116,16 +128,17 @@ end # => %ExCodecs.Codec{ # name: :zstd, # category: :compression, +# interface: :binary, # module: ExCodecs.Compression.Zstd, # native?: true, -# streaming?: true, +# streaming?: false, # configurable?: true, -# version: "1.5.6" +# version: "structured-zstd-0.0.48" # } -info.streaming? # => true +info.streaming? # => false info.configurable? # => true -info.version # => "1.5.6" +info.version # => "structured-zstd-0.0.48" ``` This is useful for checking codec capabilities: @@ -146,7 +159,9 @@ ExCodecs.Compression.available_codecs() # => [%ExCodecs.Codec{name: :blosc2, ...}, %ExCodecs.Codec{name: :bzip2, ...}, ...] ``` -Currently, all codecs are in the `:compression` category, but the architecture supports future categories (hashing, checksums, encodings). +Built-ins currently occupy `:compression` and `:spatial`. The `interface` +metadata records whether operations use the top-level binary API or the +specialized spatial API. ### Direct Lookup @@ -305,11 +320,12 @@ The registry distinguishes between "all registered codecs" and "available codecs ```elixir # All codecs known to the registry (including unavailable ones) ExCodecs.CodecRegistry.all_codecs() -# => [:blosc2, :bzip2, :lz4, :snappy, :zstd] +# => [:blosc2, :bzip2, :gsplat, :lz4, :ply, :snappy, :spatial_binary, :zstd] -# Only codecs whose NIF is loaded and functional +# Only entries whose implementation is loaded and functional ExCodecs.CodecRegistry.available_codecs() -# => [:bzip2, :lz4, :snappy, :zstd] # (blosc2 unavailable in this example) +# => [:bzip2, :gsplat, :lz4, :ply, :snappy, :spatial_binary, :zstd] +# (:blosc2 is unavailable in this example) ``` The set of available codecs may change if the NIF is reloaded. In normal operation, once the NIF loads successfully, all built-in codecs are available. diff --git a/guides/understanding_blosc2.md b/guides/understanding_blosc2.md index 947c428..a3f0e36 100644 --- a/guides/understanding_blosc2.md +++ b/guides/understanding_blosc2.md @@ -1,6 +1,17 @@ # Understanding Blosc2 -Blosc2 is a meta-compressor designed for high-performance compression of binary data, especially numerical arrays. This guide explains how Blosc2 works, its unique features (shuffle filters, internal codecs, multi-threading), and how to use it effectively in ExCodecs. +ExCodecs implements **C-Blosc2-compatible chunks** only (pure Rust via +`blosc2-pure-rs`). That means: + +| You can | You cannot (with ExCodecs alone) | +|---------|----------------------------------| +| `encode(:blosc2, binary)` → one chunk | Open/save `.b2frame` super-chunk files | +| Interop with python-blosc2 `compress`/`decompress` on a buffer | B2ND multi-dim arrays / partial ND slices | +| Use `:blosclz`, `:lz4`, `:lz4hc`, `:zstd`, `:zlib` + shuffle | `cname: :snappy` (use standalone `:snappy` codec) | + +**Standalone Snappy** remains `ExCodecs.encode(:snappy, data)`. + +Blosc2 is a meta-compressor designed for high-performance compression of binary data, especially numerical arrays. This guide explains how Blosc2 works, its shuffle filters and internal codecs, and how to use it effectively in ExCodecs. ## Overview @@ -9,11 +20,13 @@ Blosc2 is not a compression algorithm itself. It is a **meta-compressor**: a fra The key innovation in Blosc2 is combining: 1. **Shuffle filters** that reorganize bytes to improve compressibility of typed data. -2. **Pluggable internal compressors** (LZ4, LZ4HC, BloscLZ, Zstd, Snappy, Zlib). -3. **Multi-threaded compression/decompression** for parallel block processing. -4. **Block-based operation** that enables partial decompression. +2. **Pluggable internal compressors** — in ExCodecs: BloscLZ, LZ4, LZ4HC, Zstd, Zlib + (not Snappy as `cname`; use the standalone `:snappy` codec for Snappy). +3. **Block-based chunks** — ExCodecs compresses one buffer to one chunk + (`nthreads: 1` on the NIF; not a multi-chunk super-chunk store). -ExCodecs exposes Blosc2 as a first-class codec with configurable internal compressor, compression level, shuffle mode, and other parameters. +ExCodecs exposes Blosc2 as a first-class **chunk** codec with configurable +internal compressor, compression level, shuffle mode, and typesize. ## How Blosc2 Works @@ -23,7 +36,7 @@ Blosc2 processes data in a pipeline: Input Binary | v -[1] Split into blocks (blocksize or auto) +[1] Split into internally selected blocks | v [2] Apply shuffle filter (none / byte / bit) @@ -32,7 +45,7 @@ Input Binary [3] Compress each block with internal codec | v -[4] Assemble Blosc2 frame header + compressed blocks +[4] Assemble one Blosc2 chunk | v Output Binary @@ -41,10 +54,10 @@ Output Binary Decompression reverses the pipeline: ``` -Input Binary (Blosc2 frame) +Input Binary (Blosc2 chunk) | v -[1] Parse frame header +[1] Parse chunk header | v [2] Decompress each block with internal codec @@ -61,13 +74,12 @@ Output Binary (original) ### Step 1: Block Splitting -Blosc2 splits the input into blocks of configurable size. Smaller blocks enable: +Blosc2 chunks split input into internal blocks. In the wider Blosc2 ecosystem, +blocks can support parallel or partial processing. ExCodecs currently exposes +neither block-size selection nor block-level access; the implementation chooses +the block layout and decodes the complete chunk. -- Parallel processing (different blocks on different threads). -- Partial decompression (read only the blocks you need). -- Better cache utilization during compression. - -The `blocksize` parameter controls this. A value of `0` lets Blosc2 choose automatically based on the input size and number of threads. +Blocks still improve cache utilization inside compression. ### Step 2: Shuffle Filters @@ -101,25 +113,22 @@ Each (possibly shuffled) block is compressed using one of several codecs: | Codec | Description | Speed | Ratio | |------------|----------------------------------------------|---------------|-----------| -| `:blosclz` | Blosc's custom LZ-based codec | Very Fast | Moderate | +| `:blosclz` | BloscLZ (C-Blosc2 pure-Rust port) | Very Fast | Moderate | | `:lz4` | LZ4 (default) | Very Fast | Moderate | -| `:lz4hc` | LZ4 Higher Compression | Moderate | Good | -| `:snappy` | Snappy | Very Fast | Low-Moderate | +| `:lz4hc` | LZ4 high compression | Moderate | Good | | `:zstd` | Zstandard | Fast | High | | `:zlib` | Zlib (Deflate) | Moderate | Good | The choice of internal codec determines the speed/ratio tradeoff within each block. Blosc2 adds the shuffle filtering on top, so the effective ratio can be much higher than using the same codec directly. -### Step 4: Frame Assembly +### Step 4: Chunk Assembly -Blosc2 assembles the compressed blocks into a frame with a header containing: +Blosc2 assembles the compressed blocks into a chunk with metadata including: - The decompressed size. - The block size. - The internal codec, compression level, and shuffle mode. - The element typesize. -- The number of threads used. -- Checksums for data integrity. This header allows decompression without any external metadata. @@ -165,38 +174,40 @@ This header allows decompression without any external metadata. ) ``` -### Multi-Threaded Compression +### Single-threaded chunk compression ```elixir {:ok, compressed} = ExCodecs.encode(:blosc2, data, cname: :lz4, clevel: 5, shuffle: :byte, - typesize: 8, - numthreads: 4 + typesize: 8 ) ``` +The current NIF always uses one Blosc2 worker (`nthreads: 1`) on a BEAM +DirtyCpu scheduler. `:numthreads` is not a public option. + ## Configuration Options | Option | Type | Default | Description | |---------------|------------------|------------|-----------------------------------------------------| -| `:cname` | atom | `:lz4` | Internal compressor (`:lz4`, `:lz4hc`, `:blosclz`, `:zstd`, `:snappy`, `:zlib`) | +| `:cname` | atom | `:lz4` | Internal compressor (`:blosclz`, `:lz4`, `:lz4hc`, `:zstd`, `:zlib`) | | `:clevel` | integer (0-9) | 5 | Compression level (0 = no compression) | | `:shuffle` | atom | `:byte` | Shuffle filter (`:none`, `:byte`, `:bit`) | -| `:typesize` | integer (1-256) | 8 | Element size in bytes for shuffle | -| `:blocksize` | integer | 0 | Block size in bytes (0 = automatic) | -| `:numthreads`| integer | 1 | Number of threads for parallel compression | +| `:typesize` | integer (1-255) | 8 | Element size in bytes for shuffle | ### Choosing `cname` (Internal Codec) - **`:lz4`** (default): Best for speed-sensitive applications. The shuffle filter provides most of the ratio improvement. - **`:zstd`**: Best ratio when combined with shuffle. Use `:zstd` with `clevel: 5-9` for maximum compression of numerical data. -- **`:lz4hc`**: Moderate speed, better ratio than `:lz4`. Good balance. -- **`:blosclz`**: Blosc's proprietary LZ variant. Similar to `:lz4`. -- **`:snappy`**: Fastest, lowest ratio. Rarely the best choice since `:lz4` is fast and better. +- **`:lz4hc`**: Higher ratio than `:lz4`, slower compress. +- **`:blosclz`**: Blosc’s own LZ codec (included for C-Blosc2 interop). - **`:zlib`**: Moderate speed, good ratio. Available for compatibility. +`cname: :snappy` is rejected. Use the standalone registry codec +`ExCodecs.encode(:snappy, data)` instead. + ### Choosing `shuffle` - **`:byte`** (default): Best for most typed data. Groups bytes by position within each element. Recommended for float64, int32, and similar numeric types. @@ -234,45 +245,25 @@ Setting `typesize: 8` with `shuffle: :byte` on float64 arrays typically provides With `clevel: 0`, Blosc2 applies the shuffle filter but does not compress. This is useful when you want the reordering benefit without compression, or as a diagnostic to see how much shuffle alone helps. -### Choosing `numthreads` - -- **1** (default): Single-threaded. No thread overhead. Best for small data. -- **2-4**: Good for medium-sized data on multi-core systems. -- **8+**: Best for large data (100 MB+) on servers with many cores. - -Multi-threading works by splitting the input into blocks and compressing each block in a separate thread. The overhead of thread creation and synchronization means multi-threading only helps when the data is large enough to amortize this cost. +### Threading -On the BEAM, NIFs run in DirtyCpu schedulers. Blosc2's threads are native OS threads spawned by the Rust `blosc2` crate, which operate independently of BEAM schedulers. Be mindful of total system CPU usage when using high thread counts. +On the BEAM, the NIF runs on a DirtyCpu scheduler. The pure-Rust +C-Blosc2-compatible chunk implementation uses `nthreads: 1`; ExCodecs does not +create an extra native thread pool. Run independent codec calls concurrently +from separate BEAM processes when workload-level parallelism is needed. -## The Blosc2 Frame Format +Wire format is a **Blosc2 chunk** (not super-chunk / B2ND / `.b2frame`). +Fixtures under `test/fixtures/blosc2/` are produced with python-blosc2. -A Blosc2 compressed frame has the following structure: +## The Blosc2 Chunk Format -``` -+----------+-----------+---------+-----------+----------+ -| Header | Extended | Block 0 | Block 1 | Block N | -| (16+ B) | Header | | | | -+----------+-----------+---------+-----------+----------+ -``` +ExCodecs produces one C-Blosc2-compatible **chunk** per call. The chunk carries +the decompressed size, compressed size, block size, typesize, codec, level, and +filter metadata required for full-buffer decompression. -The header contains: -- Magic number and version -- Decompressed size -- Compressed size -- Block size -- Typesize -- Codec, compression level, and shuffle mode -- Number of threads -- Filter pipeline information - -Each block contains: -- Block header (compressed size, decompressed size) -- Compressed data - -This format enables: -- **Partial decompression**: Read and decompress specific blocks without processing the entire frame. -- **Multi-threaded decompression**: Decompress blocks in parallel. -- **Metadata**: All parameters needed for decompression are in the header. +The public API only performs complete chunk decompression. It does **not** +expose partial block reads, super-chunks, B2ND arrays, `.b2frame` containers, +or configurable native decompression threads. ## When to Use Blosc2 @@ -281,13 +272,13 @@ This format enables: - **Data is numerical arrays.** Float64, int32, and other typed data benefit enormously from shuffle filters. - **You need fine-grained control.** Blosc2 offers the most parameters of any codec in ExCodecs. - **Data has regular element structure.** If data length is a multiple of a known typesize, Blosc2 can exploit this. -- **You want multi-threaded compression.** Blosc2 is the only codec with multi-threading support. -- **You need partial decompression.** The block structure enables reading specific blocks. +- **You need C-Blosc2 chunk interoperability.** ExCodecs chunks interoperate + with single-buffer C/Python Blosc2 compression APIs. ### Consider Alternatives When - **Data is unstructured binary.** If there is no regular element structure, the shuffle filter provides no benefit, and Zstd or LZ4 directly may be simpler. -- **Data is small.** Blosc2's frame header overhead makes it less efficient for data under 256 bytes. +- **Data is small.** Blosc2's chunk-header overhead makes it less efficient for data under 256 bytes. - **You want simplicity.** Blosc2 has many configuration knobs. Zstd with `level: 3` is simpler and effective for most data. ## Ratio Improvement from Shuffle @@ -309,10 +300,11 @@ For numerical data, the improvement from shuffle is dramatic and consistent. The |--------------------|-------------------|----------------|----------------| | Category | Meta-compressor | Algorithm | Algorithm | | Shuffle Filters | Yes (byte, bit) | No | No | -| Inner Codecs | 6 options | N/A | N/A | -| Multi-threading | Yes | No | No | +| Inner Codecs | 5 options | N/A | N/A | +| Incremental API | No | No | No | +| Per-call threading | Single-threaded | Single-threaded| Single-threaded| | Best For | Numerical arrays | General data | Speed | -| Configuration | Extensive | Level + window | Level | +| Configuration | Codec/level/shuffle/typesize | Level 1-22 | None | ## Best Practices @@ -324,7 +316,8 @@ For numerical data, the improvement from shuffle is dramatic and consistent. The 4. **Start with `cname: :lz4, clevel: 5` for speed, or `cname: :zstd, clevel: 5` for ratio.** These are the most commonly useful configurations. -5. **Use `numthreads > 1` only for data over 1 MB.** Thread overhead makes multi-threading counterproductive for small data. +5. **Parallelize independent calls with BEAM processes.** Each NIF call is + single-threaded and runs on a DirtyCpu scheduler. 6. **Benchmark with your actual data.** The effectiveness of shuffle depends heavily on the data. Always measure, do not guess. diff --git a/guides/understanding_lz4.md b/guides/understanding_lz4.md index 15bc076..55ad408 100644 --- a/guides/understanding_lz4.md +++ b/guides/understanding_lz4.md @@ -9,7 +9,7 @@ LZ4 was created by Yann Collet (the same author as Zstd) and is focused on extre - Compression speeds over 500 MB/s per core. - Decompression speeds over 2 GB/s per core. - A simple, well-defined block format. -- Configurable compression levels (1-16) via LZ4 HC. +- A fixed fast-compression profile in ExCodecs. LZ4 achieves this speed by using a simple hash table for match finding and avoiding computationally expensive entropy coding. The tradeoff is a lower compression ratio compared to Zstd or Bzip2. @@ -36,29 +36,18 @@ Unlike Zstd or Deflate, LZ4 does **not** perform: This simplicity is what makes LZ4 fast. The compressed data is essentially a stream of "copy these literal bytes, then copy `length` bytes from `offset` back." -## Compression Levels +## Configuration -With the `lz4_flex` Rust crate, LZ4 offers levels 1-16 through the HC (Higher Compression) variant: - -| Level | Speed | Ratio | Use Case | -|-------|------------------|----------|----------------------------------| -| 1 | Very Fast (500+ MB/s) | Moderate | Default, real-time systems | -| 4-8 | Fast (200-400 MB/s) | Good | Slightly more compression needed | -| 9-12 | Moderate | Better | When ratio matters more than speed| -| 13-16 | Slow | Best | Maximum LZ4 ratio | +The ExCodecs LZ4 codec uses `lz4_flex` fast block compression and does **not** +expose compression levels or LZ4 HC. Pass no codec-specific encode options: ```elixir -# Default (fastest) {:ok, compressed} = ExCodecs.encode(:lz4, data) - -# Higher level for better ratio -{:ok, compressed} = ExCodecs.encode(:lz4, data, level: 9) - -# Maximum LZ4 compression (still faster than Zstd level 1) -{:ok, compressed} = ExCodecs.encode(:lz4, data, level: 16) ``` -Even at level 16, LZ4 HC is typically faster than Zstd at level 1, though Zstd achieves better ratios at all levels. +If you need a configurable speed/ratio tradeoff, use Zstd. If you need an +LZ4-family codec with a higher-compression profile, Blosc2 supports +`cname: :lz4hc` and `clevel: 0..9`. ## The LZ4 Block Format @@ -95,7 +84,7 @@ This format is simple to parse, which contributes to LZ4's fast decompression sp ### Compression Speed -LZ4 at level 1 compresses at 500+ MB/s on modern hardware. This means: +LZ4's fixed fast profile can compress at 500+ MB/s on modern hardware. This means: - A 1 MB payload compresses in under 2 ms. - A 10 MB payload compresses in under 20 ms. @@ -144,10 +133,10 @@ LZ4 and Snappy occupy a similar niche (fast, low-ratio compression). Key differe | Property | LZ4 | Snappy | |-------------------|-------------------------------|-------------------------------| -| Compression Speed | ~500 MB/s (level 1) | ~500 MB/s | +| Compression Speed | ~500 MB/s (fixed fast profile)| ~500 MB/s | | Decompression Speed | ~2 GB/s | ~1.5 GB/s | | Compression Ratio | Moderate (better than Snappy)| Slightly lower | -| Configurable | Yes (16 levels) | No | +| Configurable | No | No | | Format | LZ4 block format | Snappy framework format | | Deterministic | Yes | Yes | @@ -161,14 +150,15 @@ LZ4 generally offers better ratio at comparable speeds, while Snappy has a simpl | Decompression Speed| Faster | Fast | | Ratio | Moderate | High | | Window Size | 64 KB (fixed) | Up to 8 MB+ (configurable) | -| Dictionary | No | Yes | -| Configurable | 1-16 | 1-22 | +| Dictionary | No | Not exposed by ExCodecs | +| Configurable | No | Levels 1-22 | LZ4 is the right choice when speed is the primary concern. Zstd is the right choice when ratio matters or when you need fine-grained control over the speed/ratio tradeoff. ## Best Practices -1. **Use level 1 by default.** Higher LZ4 levels improve ratio but reduce speed. If ratio matters, consider switching to Zstd instead. +1. **Use LZ4 without level options.** ExCodecs exposes its fixed fast profile. + If ratio matters, consider Zstd or Blosc2 with `cname: :lz4hc`. 2. **Measure the compression ratio on your data.** If LZ4 provides less than 1.5:1 ratio, compression may not be worth the CPU cost. diff --git a/guides/understanding_snappy.md b/guides/understanding_snappy.md index ea94bec..8691a03 100644 --- a/guides/understanding_snappy.md +++ b/guides/understanding_snappy.md @@ -185,6 +185,8 @@ In practice, LZ4 usually compresses slightly better and decompresses slightly fa 3. **Pre-allocate decompression buffers.** Snappy encodes the decompressed size in the frame header, allowing you to allocate the exact output buffer. -4. **Combine with Blosc2 for arrays.** If you need fast compression of numerical data, Blosc2 with its Snappy inner codec (`cname: :snappy`) provides both shuffle optimization and Snappy's speed. +4. **Arrays with shuffle:** use Blosc2 with `cname: :lz4` (or `:zstd`), not + `cname: :snappy` — Snappy is not a standard C-Blosc2 inner codec in ExCodecs. + For pure Snappy, use this codec: `ExCodecs.encode(:snappy, data)`. 5. **Measure on your data.** If Snappy provides less than 1.3:1 ratio, compression may not be worthwhile. Consider passing data through uncompressed. \ No newline at end of file diff --git a/guides/understanding_spatial_codecs.md b/guides/understanding_spatial_codecs.md new file mode 100644 index 0000000..0dfd2c0 --- /dev/null +++ b/guides/understanding_spatial_codecs.md @@ -0,0 +1,98 @@ +# Understanding Spatial Codecs + +Spatial is a category in the ExCodecs framework for continuous and geometric +data: point clouds and Gaussian splats. These codecs map between structured +Elixir types and interchange formats. They do not render or use NIFs — pure +Elixir. + +ExCodecs uses specialized category APIs because data shapes differ. The +top-level registry API handles binary→binary codecs; spatial struct↔format work +uses `ExCodecs.Spatial.*`. Both follow the framework's tagged-result and +structured-error conventions, and both are registered in the shared codec +catalog. + +## Domain Types + +| Type | Role | +|------|------| +| `ExCodecs.Spatial.Point` | XYZ (+ optional color, normal, attributes) | +| `ExCodecs.Spatial.PointCloud` | Collection of points + bounds/metadata | +| `ExCodecs.Spatial.Gaussian` | Position, rotation, scale, opacity, color, SH | +| `ExCodecs.Spatial.GaussianCloud` | Collection of Gaussians | +| `ExCodecs.Spatial.Bounds` | Axis-aligned bounding box | +| `ExCodecs.Spatial.Transform` | Translation / rotation / scale metadata | +| `ExCodecs.Spatial.Metadata` | Comments and free-form entries | + +## Formats + +| Format atom | Module | Use | +|-------------|--------|-----| +| `:ply` | `ExCodecs.Spatial.Codec.PLY` | ASCII/binary PLY, Gaussian PLY | +| `:spatial_binary` | `ExCodecs.Spatial.Codec.Binary` | Compact `EXCP` point clouds | +| `:gsplat` | `ExCodecs.Spatial.Codec.Gsplat` | Compact `GSPL` Gaussians | + +PLY encode options (after Spatial `format: :ply` is selected): + +- `:ply_format` or `:format` — PLY wire encoding: `:ascii` (default), + `:binary` / `:binary_le`, or `:binary_be` (not the Spatial format atom) +- `:comments` — header comments +- `:as` — on decode: `:auto`, `:point_cloud`, or `:gaussian_cloud` +- `:source` — for streams: `:auto` (default), `:file`, or `:binary` + +Spatial stream helpers materialize the full payload today (lazy enumeration of +an in-memory list). Prefer explicit `source: :file` or `source: :binary`. +With `:auto`, a binary is opened as a path only when it looks path-like (under +4 KiB, no `ply`/`EXCP`/`GSPL` magic, separator or known extension) **and** is a +regular file. Files without separators/extensions are not auto-detected. + +Wire layouts for EXCP/GSPL/PLY schema rules are frozen in +[Spatial wire formats](../docs/spatial_formats.md). + +## API + +Use the spatial category API for all spatial work: + +```elixir +alias ExCodecs.Spatial.{Point, PointCloud} + +cloud = + PointCloud.new([ + Point.new(0.0, 0.0, 0.0, color: {255, 0, 0}), + Point.new(1.0, 1.0, 0.0, color: {0, 255, 0}) + ]) + +{:ok, ply} = ExCodecs.Spatial.encode(cloud, format: :ply) +{:ok, decoded} = ExCodecs.Spatial.decode(ply, format: :ply) + +ExCodecs.Spatial.stream_decode(ply, format: :ply) +|> Enum.take(2) + +{:ok, bin} = + ExCodecs.Spatial.stream_encode(decoded.points, format: :spatial_binary) + +ExCodecs.Spatial.available_formats() +ExCodecs.Spatial.supports?(:ply) +ExCodecs.available_codecs(:spatial) +ExCodecs.codec_info(:ply) +``` + +`ExCodecs.encode/3` and `ExCodecs.decode/3` remain **binary-interface codec atom ++ binary** only. Spatial atoms are discoverable there, but operation dispatch +returns a clear error pointing at this module. + +## Examples + +Sample PLY files ship in `priv/examples/spatial/`: + +- `cube_corners.ply` — colored XYZ point cloud +- `two_gaussians.ply` — minimal Gaussian PLY + +```elixir +path = Application.app_dir(:ex_codecs, "priv/examples/spatial/cube_corners.ply") +{:ok, cloud} = ExCodecs.Spatial.decode(File.read!(path), format: :ply) +``` + +## Out of Scope + +Supercompressed SOG, spatial indexes, LOD hierarchies, and progressive +streaming belong in future libraries built on top of ExCodecs (e.g. `ex_sog`). diff --git a/guides/understanding_zstd.md b/guides/understanding_zstd.md index dfcfa19..5138e7f 100644 --- a/guides/understanding_zstd.md +++ b/guides/understanding_zstd.md @@ -1,6 +1,6 @@ # Understanding Zstd -Zstandard (Zstd) is the default codec for most use cases in ExCodecs. This guide provides a deep dive into how Zstd works, its compression levels, dictionary compression, and how to get the most out of it. +Zstandard (Zstd) is the default codec for most use cases in ExCodecs. This guide provides a deep dive into how Zstd works, its compression levels, format capabilities, and the subset exposed by ExCodecs. ## Overview @@ -20,7 +20,9 @@ Zstd compression proceeds in three phases: Zstd scans the input using a sliding window. It searches backward for the longest match to the current position and emits a `(offset, match_length)` pair. Shorter matches that are unlikely to improve compression are skipped to save time. -The window size (controlled by `window_log`) determines how far back the algorithm can reference. A 2^23 = 8 MB window is the default; larger windows improve ratio on files with distant repetition. +The sliding window size determines how far back the algorithm can reference. +ExCodecs does not currently expose a `window_log` option; the pure-Rust backend +uses its default window behaviour. ### Phase 2: Sequence Encoding @@ -76,32 +78,38 @@ General recommendations: - **Levels 10-14**: Archival, batch processing, cold storage. - **Levels 15-22**: Extreme ratio scenarios. Only use if compression time is irrelevant. -## Dictionary Compression +## Dictionary Compression (not exposed) -Zstd supports **dictionary compression** for improving ratios on small data. Normally, compression algorithms need a large enough input to build an effective dictionary. With a pre-trained dictionary, even small inputs (a few hundred bytes) can achieve significant compression. +The Zstd format supports dictionary compression for small, structurally similar +payloads, but **ExCodecs does not currently expose dictionary training, +dictionary compression, or dictionary decompression options**. -### How It Works +At the format level, dictionary workflows normally: 1. **Train** a dictionary on a representative sample of your data. 2. **Compress** each small payload using the trained dictionary. 3. **Decompress** using the same dictionary. -The dictionary is typically 8-112 KB and is stored alongside or referenced by the compressed data. It captures the common patterns of your data domain, so small payloads benefit from the dictionary's knowledge. +The dictionary is typically stored alongside or referenced by compressed data. +Both encoder and decoder must use the same dictionary. -### When to Use Dictionaries +Potential future use cases include: - Small messages (under 100 KB) that share common structure. - JSON or protocol buffers with repeated schemas. - Log entries or metrics that follow a pattern. - When you control both the compression and decompression side. -ExCodecs currently provides block-level compression and decompression. Dictionary support requires managing the dictionary bytes externally and passing them through a custom Codec module that wraps `ExCodecs.Native` calls with dictionary parameters. +The current native API has no dictionary parameters, so a custom codec cannot +obtain dictionary support merely by forwarding options to `ExCodecs.Native`. +Use another Zstd implementation if dictionaries are required today. ## Streaming Zstd supports streaming (incremental) compression and decompression. This is useful when data is too large to fit in memory or when you need to process data as it arrives. -The `streaming?: true` flag in the codec info indicates that streaming infrastructure is available in the underlying library. ExCodecs currently exposes block-level operations. Streaming support for arbitrarily large inputs may be added in future versions. +ExCodecs currently exposes **block-level** operations only (`streaming?: false`). +Streaming support for arbitrarily large inputs may be added in future versions. ## Zstd Frame Format @@ -128,7 +136,7 @@ Properties of this format: ## Window Log -The `window_log` parameter controls the maximum reference distance during compression. It is only relevant at high compression levels and for large inputs. +Window size is not configurable in ExCodecs today (no `window_log` option). - Default: automatically determined based on the level and input size. - Minimum: 10 (1 KB window). @@ -145,7 +153,7 @@ Zstd memory usage depends on the compression level and window size: | Compress L1 | ~8 MB (hash table + window) | | Compress L3 | ~8 MB | | Compress L9 | ~32 MB | -| Compress L22| ~64 MB+ (depends on window_log) | +| Compress L22| high (backend-dependent) | | Decompress | ~window size (typically 8 MB, up to 128 MB+) | On the BEAM, compression runs in a DirtyCpu NIF, so this memory is allocated outside the Erlang heap. That memory pressure still affects the system, so be aware of these numbers when running many concurrent compressions. @@ -173,6 +181,7 @@ Zstd is strictly superior to GZIP/Deflate on both ratio and speed. It is the rec 4. **Reserve levels 15-22 for batch processing.** These levels can be 5-20x slower than level 3 for modest ratio improvements (typically 2-5% additional compression). -5. **Watch memory at high levels.** Levels above 15 and large `window_log` values can consume significant memory. Monitor your system under production load. +5. **Watch memory at high levels.** Large payloads require room for input and output buffers (block API). -6. **Consider dictionary compression for small payloads.** If you are compressing many small messages with shared structure, a trained dictionary can double or triple compression ratios. \ No newline at end of file +6. **Do not pass dictionary options.** They are not implemented by ExCodecs. + Evaluate another Zstd implementation when dictionary compression is required. \ No newline at end of file diff --git a/lib/ex_codecs.ex b/lib/ex_codecs.ex index 3c566bf..3c8d685 100644 --- a/lib/ex_codecs.ex +++ b/lib/ex_codecs.ex @@ -1,77 +1,140 @@ defmodule ExCodecs do @moduledoc """ - An extensible BEAM-native codec framework for Elixir. + Extensible BEAM-native **codec framework** for Elixir. - ExCodecs provides a unified API for compression, decompression, hashing, - checksums, binary encodings, and future content-addressing codecs. + ## One framework, specialized category APIs - ## Quick Start + Binary→binary registry codecs use: - # Compression - {:ok, compressed} = ExCodecs.encode(:zstd, my_binary) - {:ok, original} = ExCodecs.decode(:zstd, compressed) + ExCodecs.encode(codec_atom, binary, opts \\\\ []) + ExCodecs.decode(codec_atom, binary, opts \\\\ []) - # With options - {:ok, compressed} = ExCodecs.encode(:zstd, my_binary, level: 3) - {:ok, compressed} = ExCodecs.encode(:blosc2, my_binary, cname: :zstd, clevel: 5, shuffle: :byte) + All implementations are registered in the shared `ExCodecs.CodecRegistry` + catalog. Category modules provide entry points suited to their data: - # Discovery - ExCodecs.available_codecs() #=> [:blosc2, :bzip2, :lz4, :snappy, :zstd] - ExCodecs.supports?(:zstd) #=> true - ExCodecs.codec_info(:zstd) #=> {:ok, %ExCodecs.Codec{...}} + * `ExCodecs.Compression` — `compress/3` / `decompress/3` aliases of the + registry API, plus listing codecs in the compression category + * `ExCodecs.Spatial` — domain types and formats for point clouds / + Gaussians (struct↔format). Call `ExCodecs.Spatial.encode/2` and + `ExCodecs.Spatial.decode/2` - ## Supported Codecs + The APIs belong to one framework and share tagged results and + `%ExCodecs.Error{}` conventions. They are not overloaded because their input + shapes differ. Registry encoding and decoding keep the codec atom first: - | Codec | Category | Description | - |----------|-------------|--------------------------------------| - | `:zstd` | compression | Zstandard - high ratio, good speed | - | `:lz4` | compression | LZ4 - extremely fast | - | `:snappy`| compression | Snappy - fast, low overhead | - | `:bzip2` | compression | Bzip2 - high ratio, slower | - | `:blosc2`| compression | Blosc2 - meta-compressor for arrays | + {:ok, compressed} = ExCodecs.encode(:zstd, data) + {:ok, decoded} = ExCodecs.decode(:zstd, compressed) - ## Design Philosophy + ## Registry codecs - ExCodecs is not a compression library. It is a codec framework. - Compression is merely the first codec category. The architecture - supports future expansion into hashing, checksums, binary encodings, - content addressing, and streaming — without changing the public API. + | Codec | Notes | + |-------|--------| + | `:zstd` | Pure-Rust Zstd (`structured-zstd`) | + | `:lz4` | Size-prepended `lz4_flex` blocks | + | `:snappy` | Standalone Snappy codec | + | `:bzip2` | Pure-Rust bzip2 | + | `:blosc2` | **C-Blosc2 chunk** (not super-chunk / B2ND / `.b2frame`) | + + ### Snappy vs Blosc2 `cname: :snappy` + + - `ExCodecs.encode(:snappy, data)` — **supported** (standalone codec). + - `ExCodecs.encode(:blosc2, data, cname: :snappy)` — **rejected** (`:invalid_options`). + Snappy is not a standard C-Blosc2 inner compressor in this build; use + `:lz4`, `:blosclz`, `:zstd`, `:lz4hc`, or `:zlib` inside Blosc2. + + ### Blosc2 “chunk only” + + `:blosc2` compresses **one buffer → one Blosc2 chunk**. That matches normal + ExCodecs use (and python-blosc2 `compress`/`decompress` on a single buffer). + It does **not** open `.b2frame` files, append super-chunks, or slice B2ND + arrays. For large data, chunk yourself and store multiple blobs. + + ## Quick start + + # Registry compression + {:ok, c} = ExCodecs.encode(:zstd, "hello") + {:ok, "hello"} = ExCodecs.decode(:zstd, c) + + # Category alias + {:ok, c} = ExCodecs.Compression.compress(:lz4, data) + + # Spatial category + alias ExCodecs.Spatial.{Point, PointCloud} + cloud = PointCloud.new([Point.new(0.0, 0.0, 0.0)]) + {:ok, ply} = ExCodecs.Spatial.encode(cloud, format: :ply) + {:ok, cloud} = ExCodecs.Spatial.decode(ply, format: :ply) + + ExCodecs.available_codecs() + #=> [:blosc2, :bzip2, :gsplat, :lz4, :ply, :snappy, :spatial_binary, :zstd] + + ## Error policy + + Public encode/decode paths return `{:ok, _}` or `{:error, %ExCodecs.Error{}}`. + They do **not** raise for invalid codecs, bad options, or compression failure. + (NIF load failure is converted to `{:error, %Error{reason: :nif_not_loaded}}`.) """ alias ExCodecs.{Codec, CodecRegistry, Error} @doc """ - Encodes data using the specified codec. + Encodes a **binary** with a **registered** codec. - For compression codecs, this compresses the data. For other codec - categories, the semantics depend on the codec type. + Primary framework entry point. First argument is always a codec atom + (`:zstd`, `:lz4`, …). For spatial structs use `ExCodecs.Spatial.encode/2`. ## Arguments - * `codec` - The codec atom (e.g., `:zstd`, `:lz4`) - * `data` - The binary data to encode - * `opts` - Codec-specific options (default: `[]`) + * `codec` (`atom()`) — registry key, such as `:zstd` + * `data` (`binary()`) — unencoded bytes + * `opts` (`keyword()`) — codec-specific options; defaults to `[]` ## Returns - * `{:ok, encoded_binary}` - Successfully encoded data - * `{:error, %ExCodecs.Error{}}` - Encoding failed + * `{:ok, binary()}` — encoded payload + * `{:error, %ExCodecs.Error{reason: :unsupported_codec}}` — `codec` is not + registered + * `{:error, %ExCodecs.Error{reason: :codec_unavailable}}` — `codec` is + registered without an implementation module + * `{:error, %ExCodecs.Error{reason: :invalid_data}}` — `data` is not a + binary, arguments do not have the documented shape, or the codec rejects + the input + * `{:error, %ExCodecs.Error{reason: :invalid_options}}` — codec-specific + option validation failed + * `{:error, %ExCodecs.Error{reason: :compression_failed}}` — native encoder + failed + * `{:error, %ExCodecs.Error{reason: :nif_not_loaded}}` — native library is + unavailable + + ## Raises + + Does not raise for the documented failure modes, including malformed public + arguments. It may raise `ArgumentError` if the registry has not been started, + or propagate an unexpected exception from a third-party registered codec. ## Examples - iex> {:ok, compressed} = ExCodecs.encode(:zstd, "hello world") - iex> is_binary(compressed) - true + iex> {:ok, compressed} = ExCodecs.encode(:zstd, "hello world") + iex> is_binary(compressed) + true - iex> {:ok, compressed} = ExCodecs.encode(:zstd, "hello world", level: 3) - iex> is_binary(compressed) - true + iex> {:ok, compressed} = ExCodecs.encode(:zstd, "hello world", level: 3) + iex> is_binary(compressed) + true + + iex> {:error, %ExCodecs.Error{reason: :unsupported_codec}} = + ...> ExCodecs.encode(:not_a_codec, "x") + iex> true + true """ @spec encode(atom(), binary(), keyword()) :: {:ok, binary()} | {:error, Error.t()} def encode(codec, data, opts \\ []) def encode(codec, data, opts) when is_atom(codec) and is_binary(data) and is_list(opts) do case CodecRegistry.lookup(codec) do + {:ok, {_module, _category, %{interface: interface}}} + when interface != :binary -> + interface_error(codec, :encode) + {:ok, {module, _category, info}} -> case ensure_available(info, codec) do :ok -> module.encode(data, opts) @@ -83,25 +146,63 @@ defmodule ExCodecs do end end + def encode(codec, _data, _opts) when is_atom(codec) do + {:error, + Error.new(:invalid_data, + codec: codec, + message: "ExCodecs.encode/3 expects a binary as the second argument" + )} + end + + def encode(%_{} = struct, _data_or_opts, _opts) do + {:error, + Error.new(:invalid_data, + message: + "Structured spatial data is encoded via ExCodecs.Spatial.encode/2 " <> + "(got #{inspect(struct.__struct__)}). Registry encode/3 is binary codecs only." + )} + end + def encode(_codec, _data, _opts) do - {:error, Error.new(:invalid_data, message: "Data must be a binary")} + {:error, + Error.new(:invalid_data, + message: "ExCodecs.encode/3 expects encode(codec_atom, binary, opts \\\\ [])" + )} end @doc """ - Decodes data using the specified codec. + Decodes a **binary** with a **registered** codec. - For compression codecs, this decompresses the data. + First argument is always a codec atom. For spatial formats use + `ExCodecs.Spatial.decode/2` with `format:`. ## Arguments - * `codec` - The codec atom (e.g., `:zstd`, `:lz4`) - * `data` - The binary data to decode - * `opts` - Codec-specific options (default: `[]`) + * `codec` (`atom()`) — registry key, such as `:zstd` + * `data` (`binary()`) — encoded payload + * `opts` (`keyword()`) — codec-specific decoding options; defaults to `[]` ## Returns - * `{:ok, decoded_binary}` - Successfully decoded data - * `{:error, %ExCodecs.Error{}}` - Decoding failed + * `{:ok, binary()}` — decoded payload + * `{:error, %ExCodecs.Error{reason: :unsupported_codec}}` — `codec` is not + registered + * `{:error, %ExCodecs.Error{reason: :codec_unavailable}}` — `codec` has no + implementation module + * `{:error, %ExCodecs.Error{reason: :invalid_data}}` — arguments have the + wrong shape or the codec rejects the input + * `{:error, %ExCodecs.Error{reason: :invalid_options}}` — options are + invalid, including the mistaken `decode(binary, format: format)` shape + * `{:error, %ExCodecs.Error{reason: :decompression_failed}}` — payload is + corrupt or the native decoder failed + * `{:error, %ExCodecs.Error{reason: :nif_not_loaded}}` — native library is + unavailable + + ## Raises + + Does not raise for the documented failure modes. It may raise `ArgumentError` + if the registry has not been started, or propagate an unexpected exception + from a third-party registered codec. ## Examples @@ -115,6 +216,10 @@ defmodule ExCodecs do def decode(codec, data, opts) when is_atom(codec) and is_binary(data) and is_list(opts) do case CodecRegistry.lookup(codec) do + {:ok, {_module, _category, %{interface: interface}}} + when interface != :binary -> + interface_error(codec, :decode) + {:ok, {module, _category, info}} -> case ensure_available(info, codec) do :ok -> module.decode(data, opts) @@ -126,14 +231,138 @@ defmodule ExCodecs do end end + def decode(codec, _data, _opts) when is_atom(codec) do + {:error, + Error.new(:invalid_data, + codec: codec, + message: "ExCodecs.decode/3 expects a binary as the second argument" + )} + end + + def decode(data, opts, []) when is_binary(data) and is_list(opts) do + if Keyword.has_key?(opts, :format) do + {:error, + Error.new(:invalid_options, + message: + "Spatial formats use ExCodecs.Spatial.decode/2 " <> + "(e.g. ExCodecs.Spatial.decode(data, format: :ply)). " <> + "Registry decode/3 is decode(codec_atom, binary, opts)." + )} + else + {:error, + Error.new(:invalid_data, + message: "ExCodecs.decode/3 expects decode(codec_atom, binary, opts \\\\ [])" + )} + end + end + def decode(_codec, _data, _opts) do - {:error, Error.new(:invalid_data, message: "Data must be a binary")} + {:error, + Error.new(:invalid_data, + message: "ExCodecs.decode/3 expects decode(codec_atom, binary, opts \\\\ [])" + )} end @doc """ - Returns a list of all available codec names. + Lazily enumerates **spatial** primitives from a file path or binary. - Only codecs that are loadable and functional are included. + Delegates to `ExCodecs.Spatial.stream_decode/2`. **Not** used for registry + compression codecs. Today this materializes the payload then streams the list. + + ## Arguments + + * `source` (`Path.t() | binary()`) — filesystem path or encoded binary; + use the `:source` option to disambiguate a binary that is also a path + * `opts` (`keyword()`) — requires `:format` (`:ply`, `:spatial_binary`, or + `:gsplat`); optionally accepts `:source` (`:auto`, `:file`, or `:binary`) + and format-specific options + + ## Returns + + An `Enumerable.t()` yielding `%ExCodecs.Spatial.Point{}` or + `%ExCodecs.Spatial.Gaussian{}` values. On failure it yields exactly one + `{:error, %ExCodecs.Error{}}` element with one of these reasons: + + * `:invalid_options` — required `:format` is absent + * `:unsupported_codec` — the format is not stream-decodable + * `:io_error` — a file source cannot be read + * `:invalid_data` — the encoded payload is malformed or truncated + + ## Raises + + Missing or invalid format values and file read errors are yielded as error + tuples. Passing non-keyword `opts` raises `FunctionClauseError`; enumeration + may propagate unexpected exceptions from the source enumerable or runtime. + + ## Examples + + iex> alias ExCodecs.Spatial.{Point, PointCloud} + iex> {:ok, bin} = ExCodecs.Spatial.encode(PointCloud.new([Point.new(1.0, 2.0, 3.0)]), format: :ply) + iex> [%Point{x: x} | _] = ExCodecs.stream_decode(bin, format: :ply) |> Enum.to_list() + iex> x == 1.0 + true + """ + @spec stream_decode(Path.t() | binary(), keyword()) :: Enumerable.t() + def stream_decode(source, opts) when is_list(opts) do + ExCodecs.Spatial.stream_decode(source, opts) + end + + @doc """ + Encodes an enumerable of spatial `%Point{}` / `%Gaussian{}` values. + + Delegates to `ExCodecs.Spatial.stream_encode/2`. Collects the enumerable + (format headers need a count). **Not** for registry compression codecs. + + ## Arguments + + * `enumerable` (`Enumerable.t()`) — points or Gaussians, all of one type + * `opts` (`keyword()`) — requires `:format` (`:ply`, `:spatial_binary`, or + `:gsplat`) and may contain format-specific options + + ## Returns + + * `{:ok, binary()}` — encoded spatial payload + * `{:error, %ExCodecs.Error{reason: :invalid_options}}` — required format + is absent or a format option is invalid + * `{:error, %ExCodecs.Error{reason: :unsupported_codec}}` — format is + unknown + * `{:error, %ExCodecs.Error{reason: :invalid_data}}` — items are mixed, + malformed, unsupported, or include an error tuple + + ## Raises + + Returns error tuples for documented format, option, and item failures. + Passing non-keyword `opts` raises `FunctionClauseError`. A value that does not + implement `Enumerable` raises `Protocol.UndefinedError`, and exceptions raised + while enumerating propagate. + + ## Examples + + iex> alias ExCodecs.Spatial.Point + iex> {:ok, bin} = ExCodecs.stream_encode([Point.new(0.0, 0.0, 0.0)], format: :ply) + iex> is_binary(bin) + true + """ + @spec stream_encode(Enumerable.t(), keyword()) :: {:ok, binary()} | {:error, Error.t()} + def stream_encode(enumerable, opts) when is_list(opts) do + ExCodecs.Spatial.stream_encode(enumerable, opts) + end + + @doc """ + Lists **registered** codec atoms that are available at runtime. + + ## Arguments + + None. + + ## Returns + + A sorted `[atom()]` containing every shared-catalog entry whose + implementation module is non-`nil`, including binary and spatial codecs. + + ## Raises + + May raise `ArgumentError` if the registry ETS table has not been started. ## Examples @@ -146,10 +375,47 @@ defmodule ExCodecs do end @doc """ - Checks if a codec is supported and available at runtime. + Lists available codec names in one category. + + ## Arguments + + * `category` — category atom such as `:compression` or `:spatial` + + ## Returns - Returns `true` only if the codec is both registered and its - native implementation is loaded. + A sorted list of available names in that category. + + ## Raises + + Raises `FunctionClauseError` for a non-atom category and may raise + `ArgumentError` if the shared catalog has not started. + + ## Examples + + iex> ExCodecs.available_codecs(:spatial) + [:gsplat, :ply, :spatial_binary] + """ + @spec available_codecs(atom()) :: [atom()] + def available_codecs(category) when is_atom(category) do + CodecRegistry.available_codecs(category) + end + + @doc """ + Returns whether a shared-catalog codec is available. + + ## Arguments + + * `codec` (`atom()`) — registry key to test + + ## Returns + + `true` if `codec` is registered with a non-`nil` implementation module; + otherwise `false`. + + ## Raises + + Raises `FunctionClauseError` if `codec` is not an atom. May raise + `ArgumentError` if the registry ETS table has not been started. ## Examples @@ -165,12 +431,23 @@ defmodule ExCodecs do end @doc """ - Returns detailed information about a codec. + Returns metadata for a registered codec. + + ## Arguments + + * `codec` (`atom()`) — registry key whose metadata is requested ## Returns - * `{:ok, %ExCodecs.Codec{}}` - Codec information - * `{:error, :unsupported_codec}` - Codec not found + * `{:ok, ExCodecs.Codec.t()}` — name, category, interface, module, + capability flags, and backend version; unavailable codecs are returned + with `module: nil` + * `{:error, :unsupported_codec}` — not registered (note: bare atom, not `%Error{}`) + + ## Raises + + Raises `FunctionClauseError` if `codec` is not an atom. May raise + `ArgumentError` if the registry ETS table has not been started. ## Examples @@ -194,4 +471,14 @@ defmodule ExCodecs do {:error, Error.new(:codec_unavailable, codec: codec)} end end + + defp interface_error(codec, operation) do + {:error, + Error.new(:invalid_options, + codec: codec, + message: + "#{inspect(codec)} uses the spatial category API; call " <> + "ExCodecs.Spatial.#{operation}/2 with format: #{inspect(codec)}" + )} + end end diff --git a/lib/ex_codecs/application.ex b/lib/ex_codecs/application.ex index 9db5bce..4c4831b 100644 --- a/lib/ex_codecs/application.ex +++ b/lib/ex_codecs/application.ex @@ -2,23 +2,80 @@ defmodule ExCodecs.Application do @moduledoc """ OTP Application callback module for ExCodecs. - Starts the `ExCodecs.CodecRegistry` and registers all built-in codecs - during application startup. + Starts the shared `ExCodecs.CodecRegistry` catalog and registers all built-in + binary and spatial codecs during application startup. """ use Application @impl true + @doc """ + Starts the ExCodecs supervision tree. + + This callback starts `ExCodecs.CodecRegistry`, registers every built-in + codec entry, and supervises the registry with a `:one_for_one` + strategy. It is invoked by OTP when the `:ex_codecs` application starts; + application code should normally use `Application.ensure_all_started/1` + instead of calling this function directly. + + ## Arguments + + * `type` — `Application.start_type()`. OTP supplies `:normal`, + `{:takeover, node}`, or `{:failover, node}`; ExCodecs does not vary its + startup behavior by type. + * `args` — startup term configured for the application. ExCodecs ignores + this value. + + ## Returns + + * `{:ok, supervisor_pid}` when the supervisor and registry start. + * `{:error, reason}` when `Supervisor.start_link/2` cannot start the + supervision tree. + * `{:ok, supervisor_pid, state}` is permitted by the + `Application.start/2` callback contract, but ExCodecs does not currently + return that form. + + ## Raises + + This callback does not deliberately raise. OTP or supervisor internals may + exit the calling process for unrecoverable startup failures. + + ## Example + + iex> {:ok, _apps} = Application.ensure_all_started(:ex_codecs) + iex> Process.whereis(ExCodecs.Supervisor) |> is_pid() + true + iex> ExCodecs.supports?(:zstd) + true + + ## Callback implementation + + An OTP application that embeds a similar registry can use the same shape: + + defmodule MyApp.Application do + use Application + + @impl Application + def start(_type, _args) do + children = [{ExCodecs.CodecRegistry, register: fn -> :ok end}] + Supervisor.start_link(children, + strategy: :one_for_one, + name: MyApp.Supervisor + ) + end + end + """ + @spec start(Application.start_type(), term()) :: + {:ok, pid()} | {:error, term()} | {:ok, pid(), term()} def start(_type, _args) do children = [ - ExCodecs.CodecRegistry + {ExCodecs.CodecRegistry, register: ®ister_all_codecs/0} ] opts = [strategy: :one_for_one, name: ExCodecs.Supervisor] case Supervisor.start_link(children, opts) do {:ok, pid} -> - register_all_codecs() {:ok, pid} error -> @@ -26,21 +83,40 @@ defmodule ExCodecs.Application do end end - defp register_all_codecs do + @doc false + def register_all_codecs do codecs = [ - {:zstd, ExCodecs.Compression.Zstd, :compression}, - {:lz4, ExCodecs.Compression.Lz4, :compression}, - {:snappy, ExCodecs.Compression.Snappy, :compression}, - {:bzip2, ExCodecs.Compression.Bzip2, :compression}, - {:blosc2, ExCodecs.Compression.Blosc2, :compression} + {:zstd, ExCodecs.Compression.Zstd, :compression, :binary, []}, + {:lz4, ExCodecs.Compression.Lz4, :compression, :binary, []}, + {:snappy, ExCodecs.Compression.Snappy, :compression, :binary, []}, + {:bzip2, ExCodecs.Compression.Bzip2, :compression, :binary, []}, + {:blosc2, ExCodecs.Compression.Blosc2, :compression, :binary, []}, + {:ply, ExCodecs.Spatial.Codec.PLY, :spatial, :spatial, + [native?: false, streaming?: false, configurable?: true, version: "PLY 1.0"]}, + {:spatial_binary, ExCodecs.Spatial.Codec.Binary, :spatial, :spatial, + [native?: false, streaming?: false, configurable?: false, version: "EXCP 1"]}, + {:gsplat, ExCodecs.Spatial.Codec.Gsplat, :spatial, :spatial, + [native?: false, streaming?: false, configurable?: false, version: "GSPL 1"]} ] - for {name, module, category} <- codecs do - if Code.ensure_loaded?(module) and function_exported?(module, :encode, 2) do - ExCodecs.CodecRegistry.register(name, module, category) + nif_ok? = ExCodecs.Native.nif_loaded?() + + for {name, module, category, interface, metadata} <- codecs do + native? = + function_exported?(module, :__codec_info__, 0) and + match?(%{native?: true}, module.__codec_info__()) + + loadable? = + Code.ensure_loaded?(module) and function_exported?(module, :encode, 2) and + function_exported?(module, :decode, 2) + + if loadable? and (not native? or nif_ok?) do + ExCodecs.CodecRegistry.register(name, module, category, interface, metadata) else - ExCodecs.CodecRegistry.register_unavailable(name, category) + ExCodecs.CodecRegistry.register_unavailable(name, category, interface) end end + + :ok end end diff --git a/lib/ex_codecs/codec.ex b/lib/ex_codecs/codec.ex index 895e018..1f9b42e 100644 --- a/lib/ex_codecs/codec.ex +++ b/lib/ex_codecs/codec.ex @@ -1,83 +1,226 @@ defmodule ExCodecs.Codec do @moduledoc """ - Behaviour definition for ExCodecs codecs. - - Every codec implementation must conform to this behaviour, providing - `encode/2` and `decode/2` callbacks that operate on binaries with - optional keyword-list configuration. - - ## Implementing a Codec - - defmodule ExCodecs.Compression.Zstd do - @behaviour ExCodecs.Codec - - @impl true - def encode(data, opts) do - level = Keyword.get(opts, :level, 3) - with :ok <- validate_level(level) do - ExCodecs.NIF.wrap(:zstd, ExCodecs.Native.zstd_compress(data, level)) - end - end - - @impl true - def decode(data, _opts) do - ExCodecs.NIF.wrap(:zstd, ExCodecs.Native.zstd_decompress(data)) - end - end - - ## Codec Metadata - - Codec modules should also export a `__codec_info__/0` function - that returns metadata about the codec for the registry: - - def __codec_info__ do - %ExCodecs.Codec{ - name: :zstd, - category: :compression, - native?: true, - streaming?: true, - configurable?: true, - version: "1.5.x" - } - end + Binary-codec behaviour and shared catalog metadata. + + Binary-interface modules registered with `ExCodecs.CodecRegistry` implement + this module's `encode/2` and `decode/2` callbacks on **binaries**. Spatial + entries share the catalog metadata struct but use the specialized + `ExCodecs.Spatial` contract because they map domain structs↔formats. + + The `%ExCodecs.Codec{}` struct describes any shared-catalog entry. Its fields + are: + + * `name` (`atom()`) — registry key, such as `:zstd` + * `category` (`atom()`) — codec category, such as `:compression` or `:spatial` + * `interface` (`:binary | :spatial`) — public API shape used by the entry + * `module` (`module() | nil`) — implementation module, or `nil` when the + codec is known but unavailable + * `native?` (`boolean()`) — whether the implementation uses a NIF + * `streaming?` (`boolean()`) — whether an incremental API is available + * `configurable?` (`boolean()`) — whether the codec accepts meaningful options + * `version` (`String.t() | nil`) — backend version, if reported + + For example: + + iex> {:ok, %ExCodecs.Codec{} = codec} = ExCodecs.codec_info(:zstd) + iex> {codec.name, codec.category, codec.module} + {:zstd, :compression, ExCodecs.Compression.Zstd} + + ## Implementing a codec + + defmodule ExCodecs.Compression.Zstd do + @behaviour ExCodecs.Codec + + @impl true + def encode(data, opts) when is_binary(data) and is_list(opts) do + # ... + end + + @impl true + def decode(data, opts) when is_binary(data) and is_list(opts) do + # ... + end + + def __codec_info__ do + %ExCodecs.Codec{ + name: :zstd, + category: :compression, + module: __MODULE__, + native?: true, + streaming?: false, + configurable?: true, + version: "structured-zstd-0.0.48" + } + end + end """ + @typedoc """ + Result returned by `c:encode/2`. + + `{:ok, encoded}` contains the encoded binary. `{:error, error}` contains an + `ExCodecs.Error` whose reason is normally `:invalid_data`, `:invalid_options`, + `:compression_failed`, or `:nif_not_loaded`. + + ## Example + + iex> {:ok, result} = ExCodecs.Compression.Zstd.encode("codec input", []) + iex> is_binary(result) + true + """ @type encode_result :: {:ok, binary()} | {:error, ExCodecs.Error.t()} + + @typedoc """ + Result returned by `c:decode/2`. + + `{:ok, decoded}` contains the decoded binary. `{:error, error}` contains an + `ExCodecs.Error` whose reason is normally `:invalid_data`, + `:invalid_options`, `:decompression_failed`, `:truncated_input`, or + `:nif_not_loaded`. + + ## Example + + iex> {:ok, encoded} = ExCodecs.Compression.Zstd.encode("codec input", []) + iex> ExCodecs.Compression.Zstd.decode(encoded, []) + {:ok, "codec input"} + """ @type decode_result :: {:ok, binary()} | {:error, ExCodecs.Error.t()} @doc """ - Encodes (compresses, hashes, etc.) the given binary data. + Encodes binary data (compress, hash, etc.). ## Arguments - * `data` - The binary data to encode - * `opts` - Codec-specific options as a keyword list + * `data` (`binary()`) — bytes to encode + * `opts` (`keyword()`) — codec-specific options; implementations should + validate every supported key and value ## Returns - * `{:ok, encoded_binary}` - Successfully encoded data - * `{:error, %ExCodecs.Error{}}` - Encoding failed + * `{:ok, binary()}` — encoded bytes + * `{:error, %ExCodecs.Error{reason: :invalid_data}}` — invalid input + * `{:error, %ExCodecs.Error{reason: :invalid_options}}` — unsupported or + invalid options + * `{:error, %ExCodecs.Error{reason: :compression_failed}}` — backend + encoding failure + * `{:error, %ExCodecs.Error{reason: :nif_not_loaded}}` — native library + unavailable + + ## Raises + + Implementations must return an error tuple for invalid data, invalid options, + and expected backend failures. They may raise only for programmer errors or + unexpected runtime faults not represented by `ExCodecs.Error`. + + ## Implementation example + + defmodule Example.ReverseCodec do + @behaviour ExCodecs.Codec + + @impl true + def encode(data, opts) when is_binary(data) and opts == [] do + {:ok, String.reverse(data)} + end + + def encode(data, _opts) when not is_binary(data) do + ExCodecs.Error.error(:invalid_data, codec: :reverse) + end + + def encode(_data, _opts) do + ExCodecs.Error.error(:invalid_options, codec: :reverse) + end + + @impl true + def decode(data, opts), do: encode(data, opts) + end """ @callback encode(data :: binary(), opts :: keyword()) :: encode_result() @doc """ - Decodes (decompresses, etc.) the given binary data. + Decodes binary data (decompress, etc.). ## Arguments - * `data` - The binary data to decode - * `opts` - Codec-specific options as a keyword list + * `data` (`binary()`) — encoded bytes to decode + * `opts` (`keyword()`) — codec-specific decoding options ## Returns - * `{:ok, decoded_binary}` - Successfully decoded data - * `{:error, %ExCodecs.Error{}}` - Decoding failed + * `{:ok, binary()}` — decoded bytes + * `{:error, %ExCodecs.Error{reason: :invalid_data}}` — input has the wrong + type or shape + * `{:error, %ExCodecs.Error{reason: :invalid_options}}` — unsupported or + invalid options + * `{:error, %ExCodecs.Error{reason: :decompression_failed}}` — malformed + payload or backend decoding failure + * `{:error, %ExCodecs.Error{reason: :truncated_input}}` — incomplete input, + when the implementation distinguishes it + * `{:error, %ExCodecs.Error{reason: :nif_not_loaded}}` — native library + unavailable + + ## Raises + + Implementations must return an error tuple for malformed data, invalid + options, and expected backend failures. They may raise only for programmer + errors or unexpected runtime faults not represented by `ExCodecs.Error`. + + ## Implementation example + + defmodule Example.PrefixCodec do + @behaviour ExCodecs.Codec + + @impl true + def encode(data, []) when is_binary(data), do: {:ok, <<"EX", data::binary>>} + + def encode(data, _opts) when not is_binary(data) do + ExCodecs.Error.error(:invalid_data, codec: :prefix) + end + + def encode(_data, _opts) do + ExCodecs.Error.error(:invalid_options, codec: :prefix) + end + + @impl true + def decode(<<"EX", data::binary>>, []), do: {:ok, data} + + def decode(data, _opts) when not is_binary(data) do + ExCodecs.Error.error(:invalid_data, codec: :prefix) + end + + def decode(_data, []), do: ExCodecs.Error.error(:decompression_failed, codec: :prefix) + def decode(_data, _opts), do: ExCodecs.Error.error(:invalid_options, codec: :prefix) + end """ @callback decode(data :: binary(), opts :: keyword()) :: decode_result() + @typedoc """ + Metadata for one shared codec-catalog entry. + + Every field is public: + + * `name` — catalog atom; binary entries can be passed to + `ExCodecs.encode/3` and `ExCodecs.decode/3` + * `category` — grouping atom used by + `ExCodecs.CodecRegistry.codecs_by_category/1` + * `interface` — `:binary` for the top-level registry API or `:spatial` for + `ExCodecs.Spatial` + * `module` — callback implementation, or `nil` for an unavailable codec + * `native?` — indicates NIF-backed operation + * `streaming?` — indicates incremental processing support + * `configurable?` — indicates codec-specific option support + * `version` — backend version string, or `nil` if unknown + + ## Example + + iex> {:ok, codec} = ExCodecs.codec_info(:zstd) + iex> %ExCodecs.Codec{name: :zstd, configurable?: configurable?} = codec + iex> is_boolean(configurable?) + true + """ @type t :: %__MODULE__{ name: atom(), category: atom(), + interface: :binary | :spatial, module: module() | nil, native?: boolean(), streaming?: boolean(), @@ -85,10 +228,39 @@ defmodule ExCodecs.Codec do version: String.t() | nil } - defstruct [:name, :category, :module, :native?, :streaming?, :configurable?, :version] + defstruct name: nil, + category: nil, + interface: :binary, + module: nil, + native?: nil, + streaming?: nil, + configurable?: nil, + version: nil @doc """ - Validates that a module implements the ExCodecs.Codec behaviour. + Returns whether `module` exports `encode/2` and `decode/2`. + + ## Arguments + + * `module` (`module()`) — module to load and inspect + + ## Returns + + * `true` — `module` loads and exports both `encode/2` and `decode/2` + * `false` — it cannot be loaded or either callback is missing + + ## Raises + + Does not raise for a valid `module()` atom. Passing a value outside the + declared type may raise `FunctionClauseError` from the code-loading API. + + ## Examples + + iex> ExCodecs.Codec.validates?(ExCodecs.Compression.Zstd) + true + + iex> ExCodecs.Codec.validates?(Nonexistent.Module) + false """ @spec validates?(module()) :: boolean() def validates?(module) do diff --git a/lib/ex_codecs/codec_registry.ex b/lib/ex_codecs/codec_registry.ex index ec38313..cfd6b0a 100644 --- a/lib/ex_codecs/codec_registry.ex +++ b/lib/ex_codecs/codec_registry.ex @@ -1,25 +1,19 @@ defmodule ExCodecs.CodecRegistry do @moduledoc """ - Runtime codec registry for ExCodecs. + Shared runtime catalog of codec implementations (ETS-backed). - The registry maintains a mapping of codec names to their implementations - and metadata. Codecs are registered at application startup and can be - queried at runtime. + Populated at application start (and again if this process restarts). Entries + carry a category and interface shape: binary codecs use the top-level + registry API, while spatial entries are dispatched through + `ExCodecs.Spatial`. - ## Registry Operations + ## Typical use iex> ExCodecs.available_codecs() - [:blosc2, :bzip2, :lz4, :snappy, :zstd] + [:blosc2, :bzip2, :gsplat, :lz4, :ply, :snappy, :spatial_binary, :zstd] iex> ExCodecs.supports?(:zstd) true - - iex> {:ok, info} = ExCodecs.codec_info(:zstd) - iex> info.name - :zstd - - The registry is backed by an ETS table for fast lookups and is - populated when the application starts. """ use Agent @@ -27,33 +21,153 @@ defmodule ExCodecs.CodecRegistry do @table_name :ex_codecs_registry @registry_name __MODULE__ + # Explicit override of the `use Agent`-generated child_spec/1 so it is + # excluded from the public-doc surface checked by DocumentationTest. The + # returned spec is identical to the generated default. + @doc false + def child_spec(arg) do + %{ + id: __MODULE__, + start: {__MODULE__, :start_link, [arg]} + } + end + @doc """ - Starts the registry agent. + Starts the registry Agent and optional re-registration callback. + + ## Arguments + + * `opts` (`keyword()`) — accepts `:register`, a zero-arity function invoked + after the named ETS table is created or cleared; it defaults to + `fn -> :ok end` + + ## Returns + + * `{:ok, pid()}` — the named registry Agent started and initialization ran + * `{:error, {:already_started, pid()}}` — the registry is already running + * `{:error, reason}` — Agent startup or the registration callback failed + + ## Raises + + With a valid keyword list, startup failures are returned by `Agent`. Passing + a non-keyword value can raise `FunctionClauseError` while reading options. + + ## Example + + A supervision tree normally starts the registry and supplies the callback: + + children = [ + {ExCodecs.CodecRegistry, register: &MyApp.Codecs.register_all/0} + ] + + Supervisor.start_link(children, strategy: :one_for_one) """ @spec start_link(keyword()) :: Agent.on_start() - def start_link(_opts \\ []) do - Agent.start_link(fn -> :ets.new(@table_name, [:set, :public, :named_table]) end, + def start_link(opts \\ []) do + register_fun = Keyword.get(opts, :register, fn -> :ok end) + + Agent.start_link( + fn -> + case :ets.whereis(@table_name) do + :undefined -> + :ets.new(@table_name, [:set, :public, :named_table]) + + _tid -> + :ets.delete_all_objects(@table_name) + end + + register_fun.() + @table_name + end, name: @registry_name ) end @doc """ - Registers a codec with the registry. + Registers a binary-interface module that exports `encode/2` and `decode/2`. ## Arguments - * `name` - The atom name of the codec (e.g., `:zstd`) - * `module` - The module implementing `ExCodecs.Codec` - * `category` - The codec category (e.g., `:compression`) + * `name` (`atom()`) — registry key used for lookups + * `module` (`module()`) — loaded module exporting `encode/2` and `decode/2` + * `category` (`atom()`) — grouping key, such as `:compression` ## Returns - * `:ok` - Codec registered successfully - * `{:error, reason}` - Registration failed + * `:ok` — metadata was stored, replacing any entry with the same `name` + * `{:error, {:invalid_codec_module, module}}` — the module cannot be loaded + or does not export both codec callbacks + + ## Raises + + May raise `ArgumentError` if the registry ETS table does not exist. A codec's + optional public `__codec_info__/0` function is called during registration and + any exception it raises propagates. + + ## Examples + + iex> :ok = ExCodecs.CodecRegistry.register( + ...> :zstd, + ...> ExCodecs.Compression.Zstd, + ...> :compression + ...> ) + iex> ExCodecs.CodecRegistry.supports?(:zstd) + true """ @spec register(atom(), module(), atom()) :: :ok | {:error, term()} def register(name, module, category) do - codec_info = build_codec_info(name, module, category) + register(name, module, category, :binary) + end + + @doc """ + Registers a module in the shared catalog with an explicit interface. + + `:binary` entries are dispatched by `ExCodecs.encode/3` / `decode/3`. + `:spatial` entries are discoverable from the same catalog but dispatched by + `ExCodecs.Spatial`. + + ## Arguments + + * `name` — catalog key + * `module` — module exporting `encode/2` and `decode/2` + * `category` — grouping atom such as `:compression` or `:spatial` + * `interface` — `:binary` or `:spatial` + + ## Returns + + `:ok` on registration or `{:error, {:invalid_codec_module, module}}`. + + ## Raises + + May raise `ArgumentError` if the registry table has not started, or propagate + an exception from a module's optional `__codec_info__/0`. + + ## Examples + + iex> :ok = ExCodecs.CodecRegistry.register( + ...> :documented_ply, + ...> ExCodecs.Spatial.Codec.PLY, + ...> :spatial, + ...> :spatial + ...> ) + iex> {:ok, info} = ExCodecs.CodecRegistry.codec_info(:documented_ply) + iex> {info.category, info.interface} + {:spatial, :spatial} + iex> ExCodecs.CodecRegistry.unregister(:documented_ply) + :ok + """ + @spec register(atom(), module(), atom(), :binary | :spatial) :: :ok | {:error, term()} + def register(name, module, category, interface) + when interface in [:binary, :spatial] do + register(name, module, category, interface, []) + end + + @doc false + @spec register(atom(), module(), atom(), :binary | :spatial, keyword()) :: + :ok | {:error, term()} + def register(name, module, category, interface, metadata) + when interface in [:binary, :spatial] and is_list(metadata) do + codec_info = build_codec_info(name, module, category, interface, metadata) case ExCodecs.Codec.validates?(module) do true -> @@ -66,13 +180,43 @@ defmodule ExCodecs.CodecRegistry do end @doc """ - Registers a codec as unavailable (known but not loadable). + Registers a known codec with `module: nil` (unavailable). + + ## Arguments + + * `name` (`atom()`) — known codec's registry key + * `category` (`atom()`) — grouping key, such as `:compression` + + ## Returns + + Always `:ok` after storing a `%ExCodecs.Codec{module: nil}` entry. + + ## Raises + + May raise `ArgumentError` if the registry ETS table does not exist. + + ## Example + + iex> :ok = ExCodecs.CodecRegistry.register_unavailable(:example_optional, :compression) + iex> {:ok, info} = ExCodecs.CodecRegistry.codec_info(:example_optional) + iex> {info.module, ExCodecs.CodecRegistry.supports?(:example_optional)} + {nil, false} + iex> ExCodecs.CodecRegistry.unregister(:example_optional) + :ok """ @spec register_unavailable(atom(), atom()) :: :ok def register_unavailable(name, category) do + register_unavailable(name, category, :binary) + end + + @doc false + @spec register_unavailable(atom(), atom(), :binary | :spatial) :: :ok + def register_unavailable(name, category, interface) + when interface in [:binary, :spatial] do codec_info = %ExCodecs.Codec{ name: name, category: category, + interface: interface, module: nil, native?: false, streaming?: false, @@ -85,15 +229,60 @@ defmodule ExCodecs.CodecRegistry do end @doc """ - Looks up a codec by name. + Deletes a registry entry (tests / hot reload). + + ## Arguments + + * `name` (`atom()`) — registry key to remove ## Returns - * `{:ok, {module, category, info}}` - Codec found - * `{:error, :unsupported_codec}` - Codec not found + Always `:ok`, whether or not an entry existed. + + ## Raises + + Raises `FunctionClauseError` when `name` is not an atom. May raise + `ArgumentError` if the registry ETS table does not exist. + + ## Example + + iex> :ok = ExCodecs.CodecRegistry.register_unavailable(:temporary_codec, :compression) + iex> :ok = ExCodecs.CodecRegistry.unregister(:temporary_codec) + iex> ExCodecs.CodecRegistry.lookup(:temporary_codec) + {:error, :unsupported_codec} + """ + @spec unregister(atom()) :: :ok + def unregister(name) when is_atom(name) do + :ets.delete(@table_name, name) + :ok + end + + @doc """ + Looks up `{module, category, info}` for `name`. + + ## Arguments + + * `name` (`atom()`) — codec registry key + + ## Returns + + * `{:ok, {module() | nil, atom(), ExCodecs.Codec.t()}}` — implementation, + category, and metadata for a registered name + * `{:error, :unsupported_codec}` — no entry exists for `name` + + ## Raises + + Raises `FunctionClauseError` when `name` is not an atom. May raise + `ArgumentError` if the registry ETS table does not exist. + + ## Example + + iex> {:ok, {module, :compression, info}} = ExCodecs.CodecRegistry.lookup(:zstd) + iex> {module, info.name} + {ExCodecs.Compression.Zstd, :zstd} """ @spec lookup(atom()) :: - {:ok, {module(), atom(), ExCodecs.Codec.t()}} | {:error, :unsupported_codec} + {:ok, {module() | nil, atom(), ExCodecs.Codec.t()}} | {:error, :unsupported_codec} def lookup(name) when is_atom(name) do case :ets.lookup(@table_name, name) do [{^name, {module, category, info}}] -> @@ -105,7 +294,26 @@ defmodule ExCodecs.CodecRegistry do end @doc """ - Returns a list of all available codec names. + Sorted list of codec atoms with a non-nil module. + + ## Arguments + + None. + + ## Returns + + A sorted `[atom()]`. Unavailable entries registered with + `register_unavailable/2` are omitted. + + ## Raises + + May raise `ArgumentError` if the registry ETS table does not exist. + + ## Example + + iex> codecs = ExCodecs.CodecRegistry.available_codecs() + iex> codecs == Enum.sort(codecs) and :zstd in codecs + true """ @spec available_codecs() :: [atom()] def available_codecs do @@ -116,7 +324,57 @@ defmodule ExCodecs.CodecRegistry do end @doc """ - Returns all registered codec names (including unavailable ones). + Sorted available catalog names in one category. + + ## Arguments + + * `category` — category atom such as `:compression` or `:spatial` + + ## Returns + + Available names with non-`nil` modules, sorted by name. + + ## Raises + + May raise `ArgumentError` if the registry table has not started. + + ## Examples + + iex> ExCodecs.CodecRegistry.available_codecs(:spatial) + [:gsplat, :ply, :spatial_binary] + """ + @spec available_codecs(atom()) :: [atom()] + def available_codecs(category) when is_atom(category) do + :ets.tab2list(@table_name) + |> Enum.filter(fn {_name, {module, cat, _info}} -> + module != nil and cat == category + end) + |> Enum.map(fn {name, _} -> name end) + |> Enum.sort() + end + + @doc """ + Sorted list of all registered names (including unavailable). + + ## Arguments + + None. + + ## Returns + + A sorted `[atom()]`, including entries whose module is `nil`. + + ## Raises + + May raise `ArgumentError` if the registry ETS table does not exist. + + ## Example + + iex> :ok = ExCodecs.CodecRegistry.register_unavailable(:documented_optional, :compression) + iex> :documented_optional in ExCodecs.CodecRegistry.all_codecs() + true + iex> ExCodecs.CodecRegistry.unregister(:documented_optional) + :ok """ @spec all_codecs() :: [atom()] def all_codecs do @@ -126,7 +384,28 @@ defmodule ExCodecs.CodecRegistry do end @doc """ - Checks if a codec is supported and available. + Returns `true` if registered and `module != nil`. + + ## Arguments + + * `name` (`atom()`) — registry key to test + + ## Returns + + `true` only when `name` is registered with a non-`nil` module; otherwise + `false`. + + ## Raises + + Raises `FunctionClauseError` when `name` is not an atom. May raise + `ArgumentError` if the registry ETS table does not exist. + + ## Example + + iex> ExCodecs.CodecRegistry.supports?(:zstd) + true + iex> ExCodecs.CodecRegistry.supports?(:unknown_codec) + false """ @spec supports?(atom()) :: boolean() def supports?(name) when is_atom(name) do @@ -137,7 +416,29 @@ defmodule ExCodecs.CodecRegistry do end @doc """ - Returns detailed information about a codec. + Returns `%ExCodecs.Codec{}` for `name`. + + ## Arguments + + * `name` (`atom()`) — codec registry key + + ## Returns + + * `{:ok, ExCodecs.Codec.t()}` — metadata for a registered codec, including + unavailable codecs + * `{:error, :unsupported_codec}` — no registry entry exists + + ## Raises + + Raises `FunctionClauseError` when `name` is not an atom. May raise + `ArgumentError` if the registry ETS table does not exist. + + ## Example + + iex> {:ok, %ExCodecs.Codec{name: :zstd, category: :compression}} = + ...> ExCodecs.CodecRegistry.codec_info(:zstd) + iex> true + true """ @spec codec_info(atom()) :: {:ok, ExCodecs.Codec.t()} | {:error, :unsupported_codec} def codec_info(name) when is_atom(name) do @@ -148,7 +449,29 @@ defmodule ExCodecs.CodecRegistry do end @doc """ - Returns all codecs in a given category. + Lists `%ExCodecs.Codec{}` for a category atom (e.g. `:compression`). + + ## Arguments + + * `category` (`atom()`) — category to select, such as `:compression` + + ## Returns + + A `[ExCodecs.Codec.t()]` sorted by each struct's `name`. The list includes + unavailable entries in that category and is empty when none match. + + ## Raises + + Raises `FunctionClauseError` when `category` is not an atom. May raise + `ArgumentError` if the registry ETS table does not exist. + + ## Example + + iex> codecs = ExCodecs.CodecRegistry.codecs_by_category(:compression) + iex> Enum.all?(codecs, &match?(%ExCodecs.Codec{category: :compression}, &1)) + true + iex> Enum.map(codecs, & &1.name) == Enum.sort(Enum.map(codecs, & &1.name)) + true """ @spec codecs_by_category(atom()) :: [ExCodecs.Codec.t()] def codecs_by_category(category) when is_atom(category) do @@ -158,7 +481,7 @@ defmodule ExCodecs.CodecRegistry do |> Enum.sort_by(& &1.name) end - defp build_codec_info(name, module, category) do + defp build_codec_info(name, module, category, interface, metadata) do info = if function_exported?(module, :__codec_info__, 0) do module.__codec_info__() @@ -169,11 +492,12 @@ defmodule ExCodecs.CodecRegistry do %ExCodecs.Codec{ name: name, category: category, + interface: interface, module: module, - native?: info.native?, - streaming?: info.streaming?, - configurable?: info.configurable?, - version: info.version + native?: Keyword.get(metadata, :native?, info.native?), + streaming?: Keyword.get(metadata, :streaming?, info.streaming?), + configurable?: Keyword.get(metadata, :configurable?, info.configurable?), + version: Keyword.get(metadata, :version, info.version) } end end diff --git a/lib/ex_codecs/compression.ex b/lib/ex_codecs/compression.ex index b4b405f..b941b83 100644 --- a/lib/ex_codecs/compression.ex +++ b/lib/ex_codecs/compression.ex @@ -1,39 +1,52 @@ defmodule ExCodecs.Compression do @moduledoc """ - Compression codec category for ExCodecs. + Compression **category module**. - This module serves as a namespace for compression codecs and provides - category-level utilities for comparing and selecting compression algorithms. + Thin category helper: domain names (`compress` / `decompress`) and listing. + The real API is still `ExCodecs.encode/3` / `decode/3` (codec atom + binary). - ## Available Codecs + ## Available codec modules - * `ExCodecs.Compression.Zstd` — High ratio, good speed - * `ExCodecs.Compression.Lz4` — Extremely fast compression - * `ExCodecs.Compression.Snappy` — Fast with low overhead - * `ExCodecs.Compression.Bzip2` — High compression ratio - * `ExCodecs.Compression.Blosc2` — Meta-compressor for array data + * `ExCodecs.Compression.Zstd` + * `ExCodecs.Compression.Lz4` + * `ExCodecs.Compression.Snappy` — standalone Snappy (not Blosc2 `cname: :snappy`) + * `ExCodecs.Compression.Bzip2` + * `ExCodecs.Compression.Blosc2` — C-Blosc2 **chunk** only - ## Compression vs Encoding - - In ExCodecs, compression is encoding — and decompression is decoding. - This consistent terminology spans all codec categories: + ## Examples - # These are equivalent - ExCodecs.encode(:zstd, data) # compress - ExCodecs.decode(:zstd, data) # decompress + {:ok, c} = ExCodecs.Compression.compress(:zstd, data, level: 3) + {:ok, d} = ExCodecs.Compression.decompress(:zstd, c) + # same as: + {:ok, c} = ExCodecs.encode(:zstd, data, level: 3) """ alias ExCodecs.CodecRegistry @doc """ - Returns all available compression codecs. + Lists metadata for every registered compression codec. + + ## Arguments + + This function takes no arguments. + + ## Returns + + A list of `ExCodecs.Codec.t()` values sorted by codec name. Each struct + describes the registry name, implementation module, native/configuration + flags, streaming support, and backend version. A known codec that is not + available at runtime may have `module: nil`. + + ## Raises / Exceptions - Returns a list of `%ExCodecs.Codec{}` structs sorted by name. + This function does not raise. ## Examples iex> codecs = ExCodecs.Compression.available_codecs() - iex> is_list(codecs) and length(codecs) >= 5 + iex> Enum.map(codecs, & &1.name) + [:blosc2, :bzip2, :lz4, :snappy, :zstd] + iex> Enum.all?(codecs, &(&1.category == :compression)) true """ @spec available_codecs() :: [ExCodecs.Codec.t()] @@ -42,9 +55,43 @@ defmodule ExCodecs.Compression do end @doc """ - Compresses data with the specified codec. + Compresses `data` with a registered compression codec. + + This is the compression-category alias for `ExCodecs.encode/3`. + + ## Arguments + + * `codec` (`atom()`) — registered compression codec, such as `:zstd`, + `:lz4`, `:snappy`, `:bzip2`, or `:blosc2` + * `data` (`binary()`) — bytes to compress + * `opts` (`keyword()`) — codec-specific options; defaults to `[]` + + ## Returns + + * `{:ok, compressed :: binary()}` on success + * `{:error, %ExCodecs.Error{reason: reason}}` on failure, where `reason` + can be: + * `:unsupported_codec` when `codec` is not registered + * `:codec_unavailable` when its implementation is unavailable + * `:invalid_data` when the arguments have the wrong shape + * `:invalid_options` when codec option validation fails + * `:compression_failed` when the native compressor rejects the operation + * `:nif_not_loaded` when the native library is unavailable + + ## Raises / Exceptions + + Normal validation and wrapped NIF failures are returned as error tuples. + Unexpected VM-level faults outside the NIF wrapper may still raise. - Shortcut for `ExCodecs.encode/3`. + ## Examples + + iex> payload = "sensor=18.4" + iex> {:ok, compressed} = ExCodecs.Compression.compress(:zstd, payload, level: 3) + iex> {:ok, ^payload} = ExCodecs.Compression.decompress(:zstd, compressed) + + iex> {:error, error} = ExCodecs.Compression.compress(:zstd, "data", level: 23) + iex> error.reason + :invalid_options """ @spec compress(atom(), binary(), keyword()) :: {:ok, binary()} | {:error, ExCodecs.Error.t()} def compress(codec, data, opts \\ []) do @@ -52,9 +99,43 @@ defmodule ExCodecs.Compression do end @doc """ - Decompresses data with the specified codec. + Decompresses `data` with a registered compression codec. + + This is the compression-category alias for `ExCodecs.decode/3`. + + ## Arguments + + * `codec` (`atom()`) — registered codec that produced the payload + * `data` (`binary()`) — compressed bytes in that codec's wire format + * `opts` (`keyword()`) — codec-specific decode options; defaults to `[]` + and is currently ignored by the built-in compression codecs + + ## Returns + + * `{:ok, decompressed :: binary()}` on success + * `{:error, %ExCodecs.Error{reason: reason}}` on failure, where `reason` + can be: + * `:unsupported_codec` when `codec` is not registered + * `:codec_unavailable` when its implementation is unavailable + * `:invalid_data` for invalid argument types or an unexpected NIF result + * `:decompression_failed` for corrupt, truncated, or mismatched input + * `:nif_not_loaded` when the native library is unavailable + + ## Raises / Exceptions + + Normal validation and wrapped NIF failures are returned as error tuples. + Unexpected VM-level faults outside the NIF wrapper may still raise. + + ## Examples + + iex> payload = <<0, 1, 2, 3, 4>> + iex> {:ok, compressed} = ExCodecs.Compression.compress(:snappy, payload) + iex> ExCodecs.Compression.decompress(:snappy, compressed) + {:ok, <<0, 1, 2, 3, 4>>} - Shortcut for `ExCodecs.decode/3`. + iex> {:error, error} = ExCodecs.Compression.decompress(:snappy, "not snappy") + iex> error.reason + :decompression_failed """ @spec decompress(atom(), binary(), keyword()) :: {:ok, binary()} | {:error, ExCodecs.Error.t()} def decompress(codec, data, opts \\ []) do diff --git a/lib/ex_codecs/compression/blosc2.ex b/lib/ex_codecs/compression/blosc2.ex index 73ef086..1bf77c4 100644 --- a/lib/ex_codecs/compression/blosc2.ex +++ b/lib/ex_codecs/compression/blosc2.ex @@ -1,29 +1,49 @@ defmodule ExCodecs.Compression.Blosc2 do @moduledoc """ - Blosc2 meta-compressor codec. + Blosc2 **chunk** codec — C-Blosc2-compatible wire format, pure Rust. - Blosc2 is a high-performance meta-compressor designed for binary data, - particularly numerical arrays. It can use other compressors (Zstd, LZ4, etc.) - internally while adding features like byte shuffle and frame-based - container formats. + ## What you get (chunk only) + + ``` + binary → [filters] → [codec] → one Blosc2 chunk binary + ``` + + This matches python-blosc2 / C-Blosc2 **single-buffer** compress/decompress. + + ### Not included + + | Layer | Status | + |-------|--------| + | Super-chunk (SChunk) | No | + | Contiguous frame (`.b2frame`) | No | + | B2ND (N-D arrays) | No | + + For large data, split buffers yourself and store multiple chunks. Usability + for “compress my array slice in Elixir” is full; for “Blosc2 as a database + file format”, use python/C Blosc2. ## Options - * `:cname` — Internal compressor: `:lz4`, `:lz4hc`, `:blosclz`, - `:zstd`, `:snappy`, `:zlib` (default: `:lz4`) - * `:clevel` — Compression level 0-9 (default: 5). 0 means no compression - (store raw data). - * `:shuffle` — Shuffle filter: `:none`, `:byte`, `:bit` (default: `:byte`). - Byte shuffle reorders data to improve compression ratios for typed data. - Bit shuffle transposes at the bit level for even better compression on numerical data. - * `:typesize` — Element size in bytes for shuffle (default: 8) + * `:cname` — `:blosclz` | `:lz4` (default) | `:lz4hc` | `:zstd` | `:zlib` + * `:clevel` — `0..9` (default `5`) + * `:shuffle` — `:none` | `:byte` (default) | `:bit` + * `:typesize` — `1..255` (default `8`) + * `:max_output_size` — Maximum allowed decompressed size in bytes + (default: 256 MiB; also hard-capped at 1 GiB per chunk) + + ## Security - ## Performance Characteristics + Do not decompress untrusted inputs without a tight `:max_output_size`. - * Optimized for numerical/array data - * Byte shuffle dramatically improves compression ratios for typed data - * Bit shuffle can further improve compression on numerical data with small deltas - * Excellent when used with appropriate typesize and shuffle settings + ### Snappy + + `cname: :snappy` is **rejected**. Use standalone `ExCodecs.encode(:snappy, data)` + or Blosc2 with `:lz4` / `:blosclz`. + + ## Implementation + + Pure-Rust `blosc2-pure-rs` (no C-Blosc2 / cmake). NIF uses single-threaded + compression (`nthreads: 1`) on DirtyCpu. ## Examples @@ -31,14 +51,6 @@ defmodule ExCodecs.Compression.Blosc2 do iex> {:ok, decompressed} = ExCodecs.decode(:blosc2, compressed) iex> decompressed <<1, 2, 3, 4, 5, 6, 7, 8>> - - iex> {:ok, compressed} = ExCodecs.encode(:blosc2, <<1, 2, 3, 4, 5, 6, 7, 8>>, cname: :zstd, clevel: 5, shuffle: :byte) - iex> is_binary(compressed) - true - - > **Note**: Blosc2 is optimized for data whose length is a multiple of - > `typesize`. For general-purpose binary compression, Zstd or LZ4 may be - > more appropriate. """ @behaviour ExCodecs.Codec @@ -48,11 +60,45 @@ defmodule ExCodecs.Compression.Blosc2 do @default_shuffle :byte @default_typesize 8 - @valid_cnames [:lz4, :lz4hc, :blosclz, :zstd, :snappy, :zlib] + @valid_cnames [:blosclz, :lz4, :lz4hc, :zstd, :zlib] @valid_shuffles [:none, :byte, :bit] @doc """ - Returns codec metadata for the registry. + Returns the registry metadata for the Blosc2 chunk codec. + + ## Arguments + + This function takes no arguments. + + ## Returns + + An `ExCodecs.Codec.t()` with these Blosc2-specific fields: + + * `name: :blosc2` and `category: :compression` + * `module: ExCodecs.Compression.Blosc2` + * `native?: true` because compression runs in a NIF + * `streaming?: false` because only individual chunks are supported + * `configurable?: true` because `encode/2` accepts compressor and filter + options + * `version: "c-blosc2-chunk/pure-rust"` for the compatible format and + backend + + ## Raises / Exceptions + + This function does not invoke the NIF and does not raise. + + ## Examples + + iex> ExCodecs.Compression.Blosc2.__codec_info__() + %ExCodecs.Codec{ + name: :blosc2, + category: :compression, + module: ExCodecs.Compression.Blosc2, + native?: true, + streaming?: false, + configurable?: true, + version: "c-blosc2-chunk/pure-rust" + } """ def __codec_info__ do %ExCodecs.Codec{ @@ -60,23 +106,67 @@ defmodule ExCodecs.Compression.Blosc2 do category: :compression, module: __MODULE__, native?: true, - streaming?: true, + streaming?: false, configurable?: true, version: blosc2_version() } end - defp blosc2_version, do: "2.x-pure-rust" + defp blosc2_version, do: "c-blosc2-chunk/pure-rust" @doc """ - Encodes (compresses) data using Blosc2. + Compresses a binary into a C-Blosc2-compatible **chunk**. - ## Options + ## Arguments + + * `data` (`binary()`) — uncompressed bytes. When shuffling, choose a + `:typesize` matching each logical value; a whole-value multiple is + preferred. + * `opts` (`keyword()`) — compression settings: + * `:cname` — `:blosclz | :lz4 | :lz4hc | :zstd | :zlib`; defaults to + `:lz4` + * `:clevel` — integer compression level in `0..9`; defaults to `5` + * `:shuffle` — `:none | :byte | :bit`; defaults to `:byte` + * `:typesize` — logical element size in bytes, integer `1..255`; + defaults to `8` + + Unknown keys are ignored. `:snappy` is not a valid `:cname`. - * `:cname` — Internal compressor atom (default: `:lz4`) - * `:clevel` — Compression level 0-9 (default: 5) - * `:shuffle` — Shuffle filter: `:none`, `:byte`, `:bit` (default: `:byte`) - * `:typesize` — Element size in bytes (default: 8) + ## Returns + + * `{:ok, chunk :: binary()}` containing one Blosc2 chunk + * `{:error, %ExCodecs.Error{reason: :invalid_data}}` when `data` is not a + binary, `opts` is not a list, or the NIF raises an argument error + * `{:error, %ExCodecs.Error{reason: :invalid_options}}` for an unsupported + `:cname` (including `:snappy`) or an out-of-range option + * `{:error, %ExCodecs.Error{reason: :compression_failed}}` when the native + compressor fails + * `{:error, %ExCodecs.Error{reason: :nif_not_loaded}}` when the native + library is unavailable + + ## Raises / Exceptions + + Guard/option validation failures and `ErlangError`/`ArgumentError` + exceptions from the NIF call are converted to error tuples. Unexpected + exception classes may propagate. + + ## Examples + + iex> samples = <<100::little-16, 101::little-16, 102::little-16>> + iex> {:ok, chunk} = + ...> ExCodecs.Compression.Blosc2.encode(samples, + ...> cname: :zstd, + ...> clevel: 7, + ...> shuffle: :byte, + ...> typesize: 2 + ...> ) + iex> ExCodecs.Compression.Blosc2.decode(chunk, []) + {:ok, <<100::little-16, 101::little-16, 102::little-16>>} + + iex> {:error, error} = + ...> ExCodecs.Compression.Blosc2.encode("data", cname: :snappy) + iex> error.reason + :invalid_options """ @impl true def encode(data, opts) when is_binary(data) and is_list(opts) do @@ -89,8 +179,7 @@ defmodule ExCodecs.Compression.Blosc2 do :ok <- validate_clevel(clevel), :ok <- validate_shuffle(shuffle), :ok <- validate_typesize(typesize) do - ExCodecs.NIF.wrap( - :blosc2, + ExCodecs.NIF.safe_call(:blosc2, fn -> ExCodecs.Native.blosc2_compress( data, cname_to_int(cname), @@ -98,20 +187,77 @@ defmodule ExCodecs.Compression.Blosc2 do shuffle_to_int(shuffle), typesize ) - ) + end) end end + def encode(_data, _opts) do + {:error, ExCodecs.Error.new(:invalid_data, codec: :blosc2)} + end + @doc """ - Decodes (decompresses) Blosc2-compressed data. + Decompresses a Blosc2 **chunk** binary. + + ## Arguments + + * `data` (`binary()`) — one chunk produced by this codec or by + C/python-blosc2 single-buffer `compress` + * `opts` (`term()`) — ignored by this direct function; callers using the + codec behaviour or registry API should pass the keyword list `[]` + + ## Returns + + * `{:ok, decompressed :: binary()}` on success + * `{:error, %ExCodecs.Error{reason: :invalid_data}}` when `data` is not a + binary, the chunk header is shorter than 16 bytes, the declared output + exceeds 1 GiB, or the NIF raises an argument error + * `{:error, %ExCodecs.Error{reason: :decompression_failed}}` when the chunk + is corrupt, truncated, unsupported, or is a Blosc2 container rather than + a single chunk + * `{:error, %ExCodecs.Error{reason: :nif_not_loaded}}` when the native + library is unavailable + + ## Raises / Exceptions + + Data guard failures and `ErlangError`/`ArgumentError` exceptions from the NIF + call are converted to error tuples. Because `opts` is ignored, this direct + function also accepts non-list option terms. Unexpected exception classes + may propagate. + + ## Examples + + iex> payload = "one Blosc2 chunk" + iex> {:ok, chunk} = + ...> ExCodecs.Compression.Blosc2.encode(payload, shuffle: :none, typesize: 1) + iex> ExCodecs.Compression.Blosc2.decode(chunk, []) + {:ok, "one Blosc2 chunk"} + + iex> {:error, error} = ExCodecs.Compression.Blosc2.decode("short", []) + iex> error.reason + :invalid_data """ @impl true - def decode(data, _opts) when is_binary(data) do - ExCodecs.NIF.wrap(:blosc2, ExCodecs.Native.blosc2_decompress(data)) + def decode(data, opts) when is_binary(data) and is_list(opts) do + with {:ok, max} <- ExCodecs.NIF.max_output_size(opts) do + ExCodecs.NIF.safe_call(:blosc2, fn -> ExCodecs.Native.blosc2_decompress(data, max) end) + end + end + + def decode(_data, _opts) do + {:error, ExCodecs.Error.new(:invalid_data, codec: :blosc2)} end defp validate_cname(cname) when cname in @valid_cnames, do: :ok + defp validate_cname(:snappy) do + {:error, + ExCodecs.Error.new(:invalid_options, + codec: :blosc2, + message: + ":snappy is not a standard C-Blosc2 compressor in this build; use one of: #{inspect(@valid_cnames)}" + )} + end + defp validate_cname(_), do: {:error, @@ -135,19 +281,18 @@ defmodule ExCodecs.Compression.Blosc2 do message: "shuffle must be one of: #{inspect(@valid_shuffles)}" )} - defp validate_typesize(ts) when is_integer(ts) and ts > 0 and ts <= 256, do: :ok + defp validate_typesize(ts) when is_integer(ts) and ts > 0 and ts <= 255, do: :ok defp validate_typesize(_), do: {:error, ExCodecs.Error.new(:invalid_options, - message: "typesize must be a positive integer up to 256" + message: "typesize must be an integer from 1 to 255" )} defp cname_to_int(:blosclz), do: 0 defp cname_to_int(:lz4), do: 1 defp cname_to_int(:lz4hc), do: 2 - defp cname_to_int(:snappy), do: 3 defp cname_to_int(:zlib), do: 4 defp cname_to_int(:zstd), do: 5 diff --git a/lib/ex_codecs/compression/bzip2.ex b/lib/ex_codecs/compression/bzip2.ex index 0648ce0..9ff37b9 100644 --- a/lib/ex_codecs/compression/bzip2.ex +++ b/lib/ex_codecs/compression/bzip2.ex @@ -1,22 +1,16 @@ defmodule ExCodecs.Compression.Bzip2 do @moduledoc """ - Bzip2 compression codec. - - Bzip2 is a high-ratio compression algorithm using the Burrows-Wheeler - block-sorting text compression algorithm and Huffman coding. It produces - smaller output than most other algorithms but is significantly slower. + Bzip2 compression codec (pure-Rust backend). ## Options - * `:block_size` — Block size multiplier, 1-9 (default: 9). Higher values - produce smaller output but use more memory. + * `:block_size` — 1..9 (default 9) + * `:max_output_size` — Maximum allowed decompressed size in bytes + (default: 256 MiB) - ## Performance Characteristics + ## Security - * Excellent compression ratio - * Slower compression and decompression than Zstd - * Higher memory usage, especially at higher block sizes - * Best for archival or storage where ratio matters more than speed + Do not decompress untrusted inputs without a tight `:max_output_size`. ## Examples @@ -35,7 +29,39 @@ defmodule ExCodecs.Compression.Bzip2 do @default_block_size 9 @doc """ - Returns codec metadata for the registry. + Returns the registry metadata for the Bzip2 codec. + + ## Arguments + + This function takes no arguments. + + ## Returns + + An `ExCodecs.Codec.t()` with these Bzip2-specific fields: + + * `name: :bzip2` and `category: :compression` + * `module: ExCodecs.Compression.Bzip2` + * `native?: true` because compression runs in a NIF + * `streaming?: false` because only complete payloads are supported + * `configurable?: true` because `encode/2` accepts `:block_size` + * `version: "bzip2-0.6/libbz2-rs"` for the backend implementation + + ## Raises / Exceptions + + This function does not invoke the NIF and does not raise. + + ## Examples + + iex> ExCodecs.Compression.Bzip2.__codec_info__() + %ExCodecs.Codec{ + name: :bzip2, + category: :compression, + module: ExCodecs.Compression.Bzip2, + native?: true, + streaming?: false, + configurable?: true, + version: "bzip2-0.6/libbz2-rs" + } """ def __codec_info__ do %ExCodecs.Codec{ @@ -49,32 +75,107 @@ defmodule ExCodecs.Compression.Bzip2 do } end - defp bzip2_version, do: "0.4.x" + defp bzip2_version, do: "bzip2-0.6/libbz2-rs" @doc """ - Encodes (compresses) data using Bzip2. + Compresses a binary with Bzip2. - ## Options + ## Arguments + + * `data` (`binary()`) — uncompressed bytes + * `opts` (`keyword()`) — options containing `:block_size`, an integer from + `1` (100 KiB blocks) through `9` (900 KiB blocks); defaults to `9`. + Unknown keys are ignored. + + ## Returns - * `:block_size` — Block size 1-9 (default: 9) + * `{:ok, compressed :: binary()}` containing a Bzip2 stream + * `{:error, %ExCodecs.Error{reason: :invalid_data}}` when `data` is not a + binary, `opts` is not a list, or the NIF raises an argument error + * `{:error, %ExCodecs.Error{reason: :invalid_options}}` when `:block_size` + is not an integer in `1..9` + * `{:error, %ExCodecs.Error{reason: :compression_failed}}` when the native + compressor fails + * `{:error, %ExCodecs.Error{reason: :nif_not_loaded}}` when the native + library is unavailable + + ## Raises / Exceptions + + Guard/option validation failures and `ErlangError`/`ArgumentError` + exceptions from the NIF call are converted to error tuples. Unexpected + exception classes may propagate. + + ## Examples + + iex> payload = :binary.copy("daily-report,", 10) + iex> {:ok, compressed} = + ...> ExCodecs.Compression.Bzip2.encode(payload, block_size: 6) + iex> is_binary(compressed) + true + iex> ExCodecs.Compression.Bzip2.decode(compressed, []) + {:ok, payload} + + iex> {:error, error} = ExCodecs.Compression.Bzip2.encode("data", block_size: 10) + iex> error.reason + :invalid_options """ @impl true def encode(data, opts) when is_binary(data) and is_list(opts) do block_size = Keyword.get(opts, :block_size, @default_block_size) with :ok <- validate_block_size(block_size) do - ExCodecs.NIF.wrap(:bzip2, ExCodecs.Native.bzip2_compress(data, block_size)) + ExCodecs.NIF.safe_call(:bzip2, fn -> ExCodecs.Native.bzip2_compress(data, block_size) end) end end + def encode(_data, _opts), do: {:error, ExCodecs.Error.new(:invalid_data, codec: :bzip2)} + @doc """ - Decodes (decompresses) Bzip2-compressed data. + Decompresses Bzip2 data. + + ## Arguments + + * `data` (`binary()`) — a complete Bzip2 stream + * `opts` (`term()`) — ignored by this direct function; callers using the + codec behaviour or registry API should pass the keyword list `[]` + + ## Returns + + * `{:ok, decompressed :: binary()}` on success + * `{:error, %ExCodecs.Error{reason: :invalid_data}}` when `data` is not a + binary or the NIF raises an argument error + * `{:error, %ExCodecs.Error{reason: :decompression_failed}}` when `data` + is corrupt, truncated, or not a Bzip2 stream + * `{:error, %ExCodecs.Error{reason: :nif_not_loaded}}` when the native + library is unavailable + + ## Raises / Exceptions + + Data guard failures and `ErlangError`/`ArgumentError` exceptions from the NIF + call are converted to error tuples. Because `opts` is ignored, this direct + function also accepts non-list option terms. Unexpected exception classes + may propagate. + + ## Examples + + iex> payload = "quarterly results" + iex> {:ok, compressed} = ExCodecs.Compression.Bzip2.encode(payload, []) + iex> ExCodecs.Compression.Bzip2.decode(compressed, []) + {:ok, "quarterly results"} + + iex> {:error, error} = ExCodecs.Compression.Bzip2.decode("not bzip2", []) + iex> error.reason + :decompression_failed """ @impl true - def decode(data, _opts) when is_binary(data) do - ExCodecs.NIF.wrap(:bzip2, ExCodecs.Native.bzip2_decompress(data)) + def decode(data, opts) when is_binary(data) and is_list(opts) do + with {:ok, max} <- ExCodecs.NIF.max_output_size(opts) do + ExCodecs.NIF.safe_call(:bzip2, fn -> ExCodecs.Native.bzip2_decompress(data, max) end) + end end + def decode(_data, _opts), do: {:error, ExCodecs.Error.new(:invalid_data, codec: :bzip2)} + defp validate_block_size(bs) when is_integer(bs) and bs >= 1 and bs <= 9, do: :ok defp validate_block_size(_), diff --git a/lib/ex_codecs/compression/lz4.ex b/lib/ex_codecs/compression/lz4.ex index c0c04ed..a3fceac 100644 --- a/lib/ex_codecs/compression/lz4.ex +++ b/lib/ex_codecs/compression/lz4.ex @@ -1,19 +1,18 @@ defmodule ExCodecs.Compression.Lz4 do @moduledoc """ - LZ4 compression codec. + LZ4 compression codec (size-prepended `lz4_flex` blocks). - LZ4 is an extremely fast compression algorithm focused on speed. - It provides compression at over 1 GB/s per core and decompression - at multi-GB/s speeds. + Not interchangeable with lz4frame / CLI `.lz4` files unless they use the + same size-prefix framing. - LZ4 does not accept configuration options. It uses a fixed compression - strategy optimized for speed. + ## Options - ## Performance Characteristics + * `:max_output_size` — Maximum allowed decompressed size in bytes + (default: 256 MiB). - * Extremely fast compression and decompression - * Lower compression ratio compared to Zstd or Bzip2 - * Ideal for real-time and latency-sensitive applications + ## Security + + Do not decompress untrusted inputs without a tight `:max_output_size`. ## Examples @@ -26,7 +25,39 @@ defmodule ExCodecs.Compression.Lz4 do @behaviour ExCodecs.Codec @doc """ - Returns codec metadata for the registry. + Returns the registry metadata for the LZ4 codec. + + ## Arguments + + This function takes no arguments. + + ## Returns + + An `ExCodecs.Codec.t()` with these LZ4-specific fields: + + * `name: :lz4` and `category: :compression` + * `module: ExCodecs.Compression.Lz4` + * `native?: true` because compression runs in a NIF + * `streaming?: false` because only complete blocks are supported + * `configurable?: false` because this codec has no options + * `version: "lz4_flex-0.11"` for the backend implementation + + ## Raises / Exceptions + + This function does not invoke the NIF and does not raise. + + ## Examples + + iex> ExCodecs.Compression.Lz4.__codec_info__() + %ExCodecs.Codec{ + name: :lz4, + category: :compression, + module: ExCodecs.Compression.Lz4, + native?: true, + streaming?: false, + configurable?: false, + version: "lz4_flex-0.11" + } """ def __codec_info__ do %ExCodecs.Codec{ @@ -40,23 +71,97 @@ defmodule ExCodecs.Compression.Lz4 do } end - defp lz4_version, do: "1.10.x" + defp lz4_version, do: "lz4_flex-0.11" @doc """ - Encodes (compresses) data using LZ4. + Compresses a binary as a size-prepended `lz4_flex` block. + + ## Arguments + + * `data` (`binary()`) — uncompressed bytes + * `opts` (`term()`) — ignored by this direct function; callers using the + codec behaviour or registry API should pass the keyword list `[]` + + ## Returns + + * `{:ok, compressed :: binary()}` containing the uncompressed-size prefix + and LZ4 block + * `{:error, %ExCodecs.Error{reason: :invalid_data}}` when `data` is not a + binary, the NIF raises an argument error, or it returns an unexpected + value + * `{:error, %ExCodecs.Error{reason: :nif_not_loaded}}` when the native + library is unavailable + + ## Raises / Exceptions - LZ4 does not accept configuration options. + Argument guard failures and `ErlangError`/`ArgumentError` exceptions from the + NIF call are converted to error tuples. Unexpected exception classes may + propagate. + + ## Examples + + iex> payload = "temperature=21.7" + iex> {:ok, compressed} = ExCodecs.Compression.Lz4.encode(payload, []) + iex> byte_size(compressed) > 4 + true + iex> ExCodecs.Compression.Lz4.decode(compressed, []) + {:ok, "temperature=21.7"} """ @impl true def encode(data, _opts) when is_binary(data) do - ExCodecs.NIF.wrap(:lz4, ExCodecs.Native.lz4_compress(data)) + ExCodecs.NIF.safe_call(:lz4, fn -> ExCodecs.Native.lz4_compress(data) end) end + def encode(_data, _opts), do: {:error, ExCodecs.Error.new(:invalid_data, codec: :lz4)} + @doc """ - Decodes (decompresses) LZ4-compressed data. + Decompresses LZ4 size-prepended data. + + ## Arguments + + * `data` (`binary()`) — a size-prepended `lz4_flex` block, normally + produced by `encode/2` + * `opts` (`keyword()`) — optional `:max_output_size` (positive integer + bytes, default 256 MiB) + + ## Returns + + * `{:ok, decompressed :: binary()}` on success + * `{:error, %ExCodecs.Error{reason: :invalid_data}}` when `data` is not a + binary, the NIF raises an argument error, or it returns an unexpected + value + * `{:error, %ExCodecs.Error{reason: :invalid_options}}` when + `:max_output_size` is not a positive integer + * `{:error, %ExCodecs.Error{reason: :output_limit_exceeded}}` when the + claimed or actual size exceeds `:max_output_size` + * `{:error, %ExCodecs.Error{reason: :decompression_failed}}` when the size + prefix or compressed block is corrupt, truncated, or incompatible + * `{:error, %ExCodecs.Error{reason: :nif_not_loaded}}` when the native + library is unavailable + + ## Raises / Exceptions + + Argument guard failures and `ErlangError`/`ArgumentError` exceptions from the + NIF call are converted to error tuples. Unexpected exception classes may + propagate. + + ## Examples + + iex> payload = <<1, 2, 3, 4, 5>> + iex> {:ok, compressed} = ExCodecs.Compression.Lz4.encode(payload, []) + iex> ExCodecs.Compression.Lz4.decode(compressed, []) + {:ok, <<1, 2, 3, 4, 5>>} + + iex> {:error, error} = ExCodecs.Compression.Lz4.decode(<<1, 2>>, []) + iex> error.reason + :decompression_failed """ @impl true - def decode(data, _opts) when is_binary(data) do - ExCodecs.NIF.wrap(:lz4, ExCodecs.Native.lz4_decompress(data)) + def decode(data, opts) when is_binary(data) and is_list(opts) do + with {:ok, max} <- ExCodecs.NIF.max_output_size(opts) do + ExCodecs.NIF.safe_call(:lz4, fn -> ExCodecs.Native.lz4_decompress(data, max) end) + end end + + def decode(_data, _opts), do: {:error, ExCodecs.Error.new(:invalid_data, codec: :lz4)} end diff --git a/lib/ex_codecs/compression/snappy.ex b/lib/ex_codecs/compression/snappy.ex index d921dee..d745c46 100644 --- a/lib/ex_codecs/compression/snappy.ex +++ b/lib/ex_codecs/compression/snappy.ex @@ -1,20 +1,19 @@ defmodule ExCodecs.Compression.Snappy do @moduledoc """ - Snappy compression codec. + Standalone Snappy compression codec. - Snappy (formerly Zippy) is a fast compression algorithm developed by Google. - It prioritizes speed over compression ratio, achieving compression speeds - of over 500 MB/s and decompression speeds over 1.5 GB/s. + This is the **registry codec** `:snappy`. It is independent of Blosc2. + Do not confuse with `ExCodecs.encode(:blosc2, data, cname: :snappy)`, which + is rejected (Snappy is not a standard C-Blosc2 inner compressor here). - Snappy does not accept configuration options. It uses a fixed compression - strategy optimized for speed. + ## Options - ## Performance Characteristics + * `:max_output_size` — Maximum allowed decompressed size in bytes + (default: 256 MiB). - * Very fast compression and decompression - * Lower compression ratio than Zstd or Bzip2 - * Minimal overhead — ideal for short-lived data - * Deterministic output for identical inputs + ## Security + + Do not decompress untrusted inputs without a tight `:max_output_size`. ## Examples @@ -27,7 +26,39 @@ defmodule ExCodecs.Compression.Snappy do @behaviour ExCodecs.Codec @doc """ - Returns codec metadata for the registry. + Returns the registry metadata for the standalone Snappy codec. + + ## Arguments + + This function takes no arguments. + + ## Returns + + An `ExCodecs.Codec.t()` with these Snappy-specific fields: + + * `name: :snappy` and `category: :compression` + * `module: ExCodecs.Compression.Snappy` + * `native?: true` because compression runs in a NIF + * `streaming?: false` because only complete buffers are supported + * `configurable?: false` because this codec has no options + * `version: "snap-1.1"` for the backend implementation + + ## Raises / Exceptions + + This function does not invoke the NIF and does not raise. + + ## Examples + + iex> ExCodecs.Compression.Snappy.__codec_info__() + %ExCodecs.Codec{ + name: :snappy, + category: :compression, + module: ExCodecs.Compression.Snappy, + native?: true, + streaming?: false, + configurable?: false, + version: "snap-1.1" + } """ def __codec_info__ do %ExCodecs.Codec{ @@ -41,23 +72,94 @@ defmodule ExCodecs.Compression.Snappy do } end - defp snappy_version, do: "1.1.x" + defp snappy_version, do: "snap-1.1" @doc """ - Encodes (compresses) data using Snappy. + Compresses a binary with Snappy. + + ## Arguments + + * `data` (`binary()`) — uncompressed bytes + * `opts` (`term()`) — ignored by this direct function; callers using the + codec behaviour or registry API should pass the keyword list `[]` + + ## Returns + + * `{:ok, compressed :: binary()}` containing a standalone Snappy block + * `{:error, %ExCodecs.Error{reason: :invalid_data}}` when `data` is not a + binary or the NIF raises an argument error + * `{:error, %ExCodecs.Error{reason: :compression_failed}}` when the native + compressor fails + * `{:error, %ExCodecs.Error{reason: :nif_not_loaded}}` when the native + library is unavailable + + ## Raises / Exceptions - Snappy does not accept configuration options. + Data guard failures and `ErlangError`/`ArgumentError` exceptions from the NIF + call are converted to error tuples. Because `opts` is ignored, this direct + function also accepts non-list option terms. Unexpected exception classes + may propagate. + + ## Examples + + iex> payload = :binary.copy("event,", 25) + iex> {:ok, compressed} = ExCodecs.Compression.Snappy.encode(payload, []) + iex> is_binary(compressed) + true + iex> ExCodecs.Compression.Snappy.decode(compressed, []) + {:ok, payload} """ @impl true def encode(data, _opts) when is_binary(data) do - ExCodecs.NIF.wrap(:snappy, ExCodecs.Native.snappy_compress(data)) + ExCodecs.NIF.safe_call(:snappy, fn -> ExCodecs.Native.snappy_compress(data) end) end + def encode(_data, _opts), do: {:error, ExCodecs.Error.new(:invalid_data, codec: :snappy)} + @doc """ - Decodes (decompresses) Snappy-compressed data. + Decompresses Snappy data. + + ## Arguments + + * `data` (`binary()`) — a standalone Snappy block, normally produced by + `encode/2` + * `opts` (`term()`) — ignored by this direct function; callers using the + codec behaviour or registry API should pass the keyword list `[]` + + ## Returns + + * `{:ok, decompressed :: binary()}` on success + * `{:error, %ExCodecs.Error{reason: :invalid_data}}` when `data` is not a + binary or the NIF raises an argument error + * `{:error, %ExCodecs.Error{reason: :decompression_failed}}` when `data` + is corrupt, truncated, or not a Snappy block + * `{:error, %ExCodecs.Error{reason: :nif_not_loaded}}` when the native + library is unavailable + + ## Raises / Exceptions + + Data guard failures and `ErlangError`/`ArgumentError` exceptions from the NIF + call are converted to error tuples. Because `opts` is ignored, this direct + function also accepts non-list option terms. Unexpected exception classes + may propagate. + + ## Examples + + iex> payload = <<10, 20, 30, 40>> + iex> {:ok, compressed} = ExCodecs.Compression.Snappy.encode(payload, []) + iex> ExCodecs.Compression.Snappy.decode(compressed, []) + {:ok, <<10, 20, 30, 40>>} + + iex> {:error, error} = ExCodecs.Compression.Snappy.decode("not snappy", []) + iex> error.reason + :decompression_failed """ @impl true - def decode(data, _opts) when is_binary(data) do - ExCodecs.NIF.wrap(:snappy, ExCodecs.Native.snappy_decompress(data)) + def decode(data, opts) when is_binary(data) and is_list(opts) do + with {:ok, max} <- ExCodecs.NIF.max_output_size(opts) do + ExCodecs.NIF.safe_call(:snappy, fn -> ExCodecs.Native.snappy_decompress(data, max) end) + end end + + def decode(_data, _opts), do: {:error, ExCodecs.Error.new(:invalid_data, codec: :snappy)} end diff --git a/lib/ex_codecs/compression/zstd.ex b/lib/ex_codecs/compression/zstd.ex index 796e5f2..8793941 100644 --- a/lib/ex_codecs/compression/zstd.ex +++ b/lib/ex_codecs/compression/zstd.ex @@ -2,22 +2,26 @@ defmodule ExCodecs.Compression.Zstd do @moduledoc """ Zstandard (Zstd) compression codec. - Zstd is a fast compression algorithm providing high compression ratios. - It was developed by Yann Collet at Facebook and offers configurable - compression levels from 1 (fastest) to 22 (smallest). + Pure-Rust backend via `structured-zstd` (no C libzstd). Compression levels + `1`–`22` are passed through to the encoder. Ratios and exact bytes may differ + from reference C Zstd at the same numeric level. ## Options - * `:level` — Compression level, 1-22 (default: 3). Higher levels produce - smaller output but take longer. - * `:window_log` — Window log size. Controls the maximum reference distance. + * `:level` — Compression level, 1-22 (default: 3). + * `:max_output_size` — Maximum allowed decompressed size in bytes + (default: 256 MiB). Rejects bombs that would expand beyond the limit. + + ## Security + + Do not decompress **untrusted** inputs without a tight `:max_output_size`. + A small malicious frame can expand to a large allocation. ## Performance Characteristics - * Fast decompression across all compression levels - * Compression speed configurable via level - * Excellent ratio at moderate speeds - * Supports dictionary compression for small data + * Pure-Rust compress/decompress (no C libzstd) + * Fast decompression + * Block-level API only (`streaming?` is `false`) ## Examples @@ -36,7 +40,39 @@ defmodule ExCodecs.Compression.Zstd do @default_level 3 @doc """ - Returns codec metadata for the registry. + Returns the registry metadata for the Zstandard codec. + + ## Arguments + + This function takes no arguments. + + ## Returns + + An `ExCodecs.Codec.t()` with these Zstd-specific fields: + + * `name: :zstd` and `category: :compression` + * `module: ExCodecs.Compression.Zstd` + * `native?: true` because compression runs in a NIF + * `streaming?: false` because only complete frames are supported + * `configurable?: true` because `encode/2` accepts `:level` + * `version: "structured-zstd-0.0.48"` for the backend implementation + + ## Raises / Exceptions + + This function does not invoke the NIF and does not raise. + + ## Examples + + iex> ExCodecs.Compression.Zstd.__codec_info__() + %ExCodecs.Codec{ + name: :zstd, + category: :compression, + module: ExCodecs.Compression.Zstd, + native?: true, + streaming?: false, + configurable?: true, + version: "structured-zstd-0.0.48" + } """ def __codec_info__ do %ExCodecs.Codec{ @@ -44,41 +80,114 @@ defmodule ExCodecs.Compression.Zstd do category: :compression, module: __MODULE__, native?: true, - streaming?: true, + streaming?: false, configurable?: true, version: zstd_version() } end - @doc false - defp zstd_version, do: "1.5.x" + defp zstd_version, do: "structured-zstd-0.0.48" @doc """ - Encodes (compresses) data using Zstd. + Compresses a binary with pure-Rust Zstd. - ## Options + ## Arguments + + * `data` (`binary()`) — uncompressed bytes + * `opts` (`keyword()`) — options containing `:level`, an integer from + `1` through `22`; the default is `3`. Unknown keys are ignored. + + ## Returns + + * `{:ok, frame :: binary()}` containing a Zstandard frame + * `{:error, %ExCodecs.Error{reason: :invalid_data}}` when `data` is not a + binary, `opts` is not a list, or the NIF raises an argument error + * `{:error, %ExCodecs.Error{reason: :invalid_options}}` when `:level` is + not an integer in `1..22` + * `{:error, %ExCodecs.Error{reason: :compression_failed}}` when the native + compressor fails + * `{:error, %ExCodecs.Error{reason: :nif_not_loaded}}` when the native + library is unavailable + + ## Raises / Exceptions + + Guard/option validation failures and `ErlangError`/`ArgumentError` + exceptions from the NIF call are converted to error tuples. Unexpected + exception classes may propagate. - * `:level` — Compression level 1-22 (default: 3) + ## Examples + + iex> payload = :binary.copy("telemetry,", 20) + iex> {:ok, compressed} = ExCodecs.Compression.Zstd.encode(payload, level: 9) + iex> is_binary(compressed) + true + iex> ExCodecs.Compression.Zstd.decode(compressed, []) + {:ok, payload} + + iex> {:error, error} = ExCodecs.Compression.Zstd.encode("data", level: 0) + iex> error.reason + :invalid_options """ @impl true def encode(data, opts) when is_binary(data) and is_list(opts) do level = Keyword.get(opts, :level, @default_level) with :ok <- validate_level(level) do - ExCodecs.NIF.wrap(:zstd, ExCodecs.Native.zstd_compress(data, level)) + ExCodecs.NIF.safe_call(:zstd, fn -> ExCodecs.Native.zstd_compress(data, level) end) end end + def encode(_data, _opts), do: {:error, ExCodecs.Error.new(:invalid_data, codec: :zstd)} + @doc """ - Decodes (decompresses) Zstd-compressed data. + Decompresses a Zstd frame binary. + + ## Arguments + + * `data` (`binary()`) — a complete Zstandard frame + * `opts` (`keyword()`) — optional `:max_output_size` (positive integer + bytes, default 256 MiB) + + ## Returns - The decompressed size is read from the frame header. + * `{:ok, decompressed :: binary()}` on success + * `{:error, %ExCodecs.Error{reason: :invalid_data}}` when `data` is not a + binary, `opts` is not a list, or the NIF raises an argument error + * `{:error, %ExCodecs.Error{reason: :invalid_options}}` when + `:max_output_size` is not a positive integer + * `{:error, %ExCodecs.Error{reason: :output_limit_exceeded}}` when the + decompressed size would exceed `:max_output_size` + * `{:error, %ExCodecs.Error{reason: :decompression_failed}}` when `data` + is corrupt, truncated, or not a Zstandard frame + * `{:error, %ExCodecs.Error{reason: :nif_not_loaded}}` when the native + library is unavailable + + ## Raises / Exceptions + + Guard failures and `ErlangError`/`ArgumentError` exceptions from the NIF + call are converted to error tuples. Unexpected exception classes may + propagate. + + ## Examples + + iex> payload = <<0, 10, 20, 30, 40>> + iex> {:ok, compressed} = ExCodecs.Compression.Zstd.encode(payload, []) + iex> ExCodecs.Compression.Zstd.decode(compressed, []) + {:ok, <<0, 10, 20, 30, 40>>} + + iex> {:error, error} = ExCodecs.Compression.Zstd.decode("not zstd", []) + iex> error.reason + :decompression_failed """ @impl true def decode(data, opts) when is_binary(data) and is_list(opts) do - ExCodecs.NIF.wrap(:zstd, ExCodecs.Native.zstd_decompress(data)) + with {:ok, max} <- ExCodecs.NIF.max_output_size(opts) do + ExCodecs.NIF.safe_call(:zstd, fn -> ExCodecs.Native.zstd_decompress(data, max) end) + end end + def decode(_data, _opts), do: {:error, ExCodecs.Error.new(:invalid_data, codec: :zstd)} + defp validate_level(level) when is_integer(level) and level >= 1 and level <= 22, do: :ok defp validate_level(_), diff --git a/lib/ex_codecs/error.ex b/lib/ex_codecs/error.ex index 16c7edb..8be5cd2 100644 --- a/lib/ex_codecs/error.ex +++ b/lib/ex_codecs/error.ex @@ -1,12 +1,64 @@ defmodule ExCodecs.Error do @moduledoc """ - Standardized error types for ExCodecs. + Structured errors for ExCodecs public APIs. - All ExCodecs functions return `{:ok, result}` or `{:error, reason}` tuples. - The `reason` will be one of the atoms defined in this module, or a structured - error when additional context is needed. + Successful operations return `{:ok, result}`. Failures return + `{:error, %ExCodecs.Error{}}` (except `codec_info/1`, which uses + `{:error, :unsupported_codec}` for historical reasons). + + `%ExCodecs.Error{}` is both the structured error value returned in tagged + tuples and an exception that can be raised explicitly. Its fields are: + + * `reason` (`error_reason()`) — stable, machine-readable reason + * `message` (`String.t()`) — human-readable exception message + * `codec` (`atom() | nil`) — related codec or format, when known + * `details` (`term() | nil`) — optional backend or I/O diagnostic data + + For example: + + iex> error = ExCodecs.Error.new(:invalid_options, codec: :zstd, details: [level: 99]) + iex> {error.reason, error.codec, error.details} + {:invalid_options, :zstd, [level: 99]} + + | Reason | Meaning | + |--------|---------| + | `:unsupported_codec` | Codec/format atom unknown | + | `:codec_unavailable` | Known but NIF/module not loadable | + | `:invalid_data` | Wrong type or corrupt payload | + | `:invalid_options` | Bad keyword options | + | `:compression_failed` | Native compress failed | + | `:decompression_failed` | Native decompress failed | + | `:nif_not_loaded` | NIF library missing | + | `:io_error` | File read/write failure | + | `:truncated_input` | Incomplete binary | + | `:output_limit_exceeded` | Decompress would exceed `max_output_size` | + + ## As exception + + `defexception` is defined so `raise error` works, but library code prefers + tagged tuples. `Exception.message/1` returns the `message` field. """ + @typedoc """ + Stable reason atom carried by `t:ExCodecs.Error.t/0`. + + * `:unsupported_codec` — codec or format is not known + * `:codec_unavailable` — codec is known but its module or NIF is unavailable + * `:invalid_data` — input has the wrong type, shape, or semantic content + * `:invalid_options` — an option name or value is invalid + * `:compression_failed` — encoding backend failed + * `:decompression_failed` — decoding backend failed or rejected corruption + * `:nif_not_loaded` — native library could not be loaded + * `:io_error` — file operation failed + * `:truncated_input` — encoded input ended before a complete value + * `:output_limit_exceeded` — decompress output would exceed `max_output_size` + + ## Example + + iex> reason = ExCodecs.Error.new(:truncated_input).reason + iex> reason in [:truncated_input, :invalid_data] + true + """ @type error_reason :: :unsupported_codec | :codec_unavailable @@ -15,7 +67,22 @@ defmodule ExCodecs.Error do | :compression_failed | :decompression_failed | :nif_not_loaded + | :io_error + | :truncated_input + | :output_limit_exceeded + + @typedoc """ + Structured ExCodecs error and exception. + The fields are `reason` (machine-readable reason), `message` (display text), + `codec` (associated codec or `nil`), and `details` (diagnostic term or `nil`). + + ## Example + + iex> %ExCodecs.Error{} = error = ExCodecs.Error.new(:io_error, details: :enoent) + iex> Exception.message(error) + "An I/O error occurred" + """ @type t :: %__MODULE__{ reason: error_reason(), message: String.t(), @@ -26,7 +93,27 @@ defmodule ExCodecs.Error do defexception [:reason, :message, :codec, :details] @doc """ - Creates a new error struct. + Builds a `%ExCodecs.Error{}`. + + ## Arguments + + * `reason` (`error_reason()`) — reason used for matching and to select the + default message + * `opts` (`keyword()`) — optional fields: + * `:message` (`String.t()`) — replaces the default message + * `:codec` (`atom() | nil`) — identifies the related codec + * `:details` (`term()`) — stores arbitrary diagnostic context + + ## Returns + + A `t:ExCodecs.Error.t/0` struct. This function never returns an error tuple. + + ## Raises + + Does not raise when called with the declared types. An improper `opts` value + can raise `FunctionClauseError`. + + ## Examples iex> error = ExCodecs.Error.new(:unsupported_codec) iex> error.reason @@ -51,7 +138,25 @@ defmodule ExCodecs.Error do end @doc """ - Creates an `{:error, ExCodecs.Error.t()}` tuple. + Builds `{:error, %ExCodecs.Error{}}`. + + ## Arguments + + * `reason` (`error_reason()`) — reason passed to `new/2` + * `opts` (`keyword()`) — message, codec, and details options accepted by + `new/2` + + ## Returns + + Always `{:error, t()}`. It has no alternative error reason because the + supplied `reason` is stored inside the struct. + + ## Raises + + Does not raise when called with the declared types. An improper `opts` value + can raise `FunctionClauseError`. + + ## Examples iex> {:error, error} = ExCodecs.Error.error(:unsupported_codec) iex> error.reason @@ -69,13 +174,58 @@ defmodule ExCodecs.Error do defp default_message(:compression_failed), do: "Compression failed" defp default_message(:decompression_failed), do: "Decompression failed" defp default_message(:nif_not_loaded), do: "The native NIF library is not loaded" + defp default_message(:io_error), do: "An I/O error occurred" + defp default_message(:truncated_input), do: "The input was truncated or incomplete" + + defp default_message(:output_limit_exceeded), + do: "Decompressed output exceeded the configured max_output_size" + defp default_message(reason), do: "Error: #{reason}" @impl true + @doc """ + Exception message string (for `raise` / `Exception.message/1`). + + ## Arguments + + * `error` (`t()`) — exception whose `message` field is returned + + ## Returns + + The `message` field as a `String.t()`. + + ## Raises + + Does not raise for a `t()` whose `message` field is a string. A value that + does not match `%ExCodecs.Error{}` raises `FunctionClauseError`. + + ## Example + + iex> error = ExCodecs.Error.new(:invalid_data, message: "not a point cloud") + iex> Exception.message(error) + "not a point cloud" + """ def message(%__MODULE__{message: message}), do: message @doc """ - Checks if an error matches a specific reason. + Returns whether an `{:error, %Error{}}` tuple matches `reason`. + + ## Arguments + + * `result` (`{:error, t()} | term()`) — result to inspect; nonmatching + terms are accepted and return `false` + * `reason` (`error_reason()`) — reason that must exactly equal the error's + `reason` field + + ## Returns + + `true` if `result` is `{:error, %Error{reason: ^reason}}`, else `false`. + + ## Raises + + None; the catch-all clause returns `false` for values of any shape. + + ## Examples iex> {:error, error} = ExCodecs.Error.error(:unsupported_codec) iex> ExCodecs.Error.matches?({:error, error}, :unsupported_codec) diff --git a/lib/ex_codecs/native.ex b/lib/ex_codecs/native.ex index f262ddb..39bbb06 100644 --- a/lib/ex_codecs/native.ex +++ b/lib/ex_codecs/native.ex @@ -38,35 +38,68 @@ defmodule ExCodecs.Native do @doc false def zstd_compress(_data, _level), do: :erlang.nif_error(:nif_not_loaded) @doc false - def zstd_decompress(_data), do: :erlang.nif_error(:nif_not_loaded) + def zstd_decompress(_data, _max_output_size), do: :erlang.nif_error(:nif_not_loaded) @doc false def lz4_compress(_data), do: :erlang.nif_error(:nif_not_loaded) @doc false - def lz4_decompress(_data), do: :erlang.nif_error(:nif_not_loaded) + def lz4_decompress(_data, _max_output_size), do: :erlang.nif_error(:nif_not_loaded) @doc false def snappy_compress(_data), do: :erlang.nif_error(:nif_not_loaded) @doc false - def snappy_decompress(_data), do: :erlang.nif_error(:nif_not_loaded) + def snappy_decompress(_data, _max_output_size), do: :erlang.nif_error(:nif_not_loaded) @doc false def bzip2_compress(_data, _block_size), do: :erlang.nif_error(:nif_not_loaded) @doc false - def bzip2_decompress(_data), do: :erlang.nif_error(:nif_not_loaded) + def bzip2_decompress(_data, _max_output_size), do: :erlang.nif_error(:nif_not_loaded) @doc false def blosc2_compress(_data, _cname, _clevel, _shuffle, _typesize), do: :erlang.nif_error(:nif_not_loaded) @doc false - def blosc2_decompress(_data), do: :erlang.nif_error(:nif_not_loaded) + def blosc2_decompress(_data, _max_output_size), do: :erlang.nif_error(:nif_not_loaded) @doc false def codec_versions, do: :erlang.nif_error(:nif_not_loaded) - @doc false - def nif_loaded?, do: not function_exported?(__MODULE__, :zstd_compress, 2) + @doc """ + Returns `true` when the native NIF library is loaded. + + The check calls the native `codec_versions/0` function and verifies that it + returns a map. This function is primarily useful for startup diagnostics; + public compression calls already convert an unavailable NIF into + `%ExCodecs.Error{reason: :nif_not_loaded}`. + + ## Arguments + + This function takes no arguments. + + ## Returns + + * `true` when the native library responds with its codec-version map. + * `false` when the NIF stub raises `ErlangError` or `ArgumentError`. + + ## Raises + + Expected NIF-loading errors are caught. An exception of another class raised + by the native implementation is not caught and propagates to the caller. + + ## Example + + iex> loaded? = ExCodecs.Native.nif_loaded?() + iex> is_boolean(loaded?) + true + """ + @spec nif_loaded?() :: boolean() + def nif_loaded? do + is_map(codec_versions()) + rescue + ErlangError -> false + ArgumentError -> false + end end # coveralls-ignore-stop diff --git a/lib/ex_codecs/nif.ex b/lib/ex_codecs/nif.ex index bcbb451..ef60dc2 100644 --- a/lib/ex_codecs/nif.ex +++ b/lib/ex_codecs/nif.ex @@ -1,15 +1,35 @@ defmodule ExCodecs.NIF do @moduledoc false + # Default decompress ceiling: 256 MiB. Override with `max_output_size:`. + @default_max_output_size 268_435_456 + + @doc false + def default_max_output_size, do: @default_max_output_size + + @doc false + def max_output_size(opts) when is_list(opts) do + case Keyword.get(opts, :max_output_size, @default_max_output_size) do + n when is_integer(n) and n > 0 -> + {:ok, n} + + _ -> + {:error, + ExCodecs.Error.new(:invalid_options, + message: "max_output_size must be a positive integer (bytes)" + )} + end + end + @doc """ Wraps raw NIF error tuples into ExCodecs.Error structs. NIF functions return `{:error, atom}` tuples. This helper converts them into `{:error, %ExCodecs.Error{}}` tuples for consistent error handling. """ - @spec wrap(atom(), {:ok, binary()} | {:error, atom()}) :: + @spec wrap(atom(), {:ok, binary()} | {:error, atom()} | term()) :: {:ok, binary()} | {:error, ExCodecs.Error.t()} - def wrap(_codec, {:ok, data}), do: {:ok, data} + def wrap(_codec, {:ok, data}) when is_binary(data), do: {:ok, data} def wrap(codec, {:error, :compression_failed}), do: {:error, ExCodecs.Error.new(:compression_failed, codec: codec)} @@ -17,11 +37,51 @@ defmodule ExCodecs.NIF do def wrap(codec, {:error, :decompression_failed}), do: {:error, ExCodecs.Error.new(:decompression_failed, codec: codec)} + def wrap(codec, {:error, :output_limit_exceeded}), + do: + {:error, + ExCodecs.Error.new(:output_limit_exceeded, + codec: codec, + message: + "Decompressed output exceeded max_output_size " <> + "(default #{@default_max_output_size} bytes). " <> + "Pass a larger max_output_size: for trusted inputs." + )} + def wrap(codec, {:error, :invalid_data}), do: {:error, ExCodecs.Error.new(:invalid_data, codec: codec)} def wrap(codec, {:error, :invalid_options}), do: {:error, ExCodecs.Error.new(:invalid_options, codec: codec)} - def wrap(codec, {:error, reason}), do: {:error, ExCodecs.Error.new(reason, codec: codec)} + def wrap(codec, {:error, reason}) when is_atom(reason), + do: {:error, ExCodecs.Error.new(reason, codec: codec)} + + def wrap(codec, other) do + {:error, + ExCodecs.Error.new(:invalid_data, + codec: codec, + message: "Unexpected NIF return: #{inspect(other)}" + )} + end + + @doc """ + Invokes a zero-arity NIF function and wraps ErlangError/`nif_not_loaded`. + """ + @spec safe_call(atom(), (-> term())) :: {:ok, term()} | {:error, ExCodecs.Error.t()} + def safe_call(codec, fun) when is_function(fun, 0) do + wrap(codec, fun.()) + rescue + e in ErlangError -> + case e.original do + :nif_not_loaded -> + {:error, ExCodecs.Error.new(:nif_not_loaded, codec: codec)} + + other -> + {:error, ExCodecs.Error.new(:invalid_data, codec: codec, details: other)} + end + + e in ArgumentError -> + {:error, ExCodecs.Error.new(:invalid_data, codec: codec, message: Exception.message(e))} + end end diff --git a/lib/ex_codecs/spatial.ex b/lib/ex_codecs/spatial.ex new file mode 100644 index 0000000..b00a12f --- /dev/null +++ b/lib/ex_codecs/spatial.ex @@ -0,0 +1,365 @@ +defmodule ExCodecs.Spatial do + @moduledoc """ + Spatial category API for point clouds and Gaussian splats. + + ExCodecs is one codec framework with entry points specialized by data shape. + Binary→binary registry codecs use `ExCodecs.encode/3` / `decode/3`; spatial + codecs map the domain structs below to and from interchange formats through + this module. + + ## Domain types + + | Struct | Role | + |--------|------| + | `ExCodecs.Spatial.Point` | XYZ (+ color, normal, attributes) | + | `ExCodecs.Spatial.PointCloud` | List of points + bounds/metadata | + | `ExCodecs.Spatial.Gaussian` | Single splat primitive | + | `ExCodecs.Spatial.GaussianCloud` | List of Gaussians | + | `ExCodecs.Spatial.Bounds` | Axis-aligned bounding box | + | `ExCodecs.Spatial.Transform` | Translation/rotation/scale metadata | + | `ExCodecs.Spatial.Metadata` | Comments and free-form entries | + + ## Formats + + | Atom | Module | Input type | + |------|--------|------------| + | `:ply` | `ExCodecs.Spatial.Codec.PLY` | PointCloud or GaussianCloud | + | `:spatial_binary` | `ExCodecs.Spatial.Codec.Binary` | PointCloud (`EXCP`) | + | `:gsplat` | `ExCodecs.Spatial.Codec.Gsplat` | GaussianCloud (`GSPL`) | + + Spatial formats are registered in the shared codec catalog. Discover all + entries with `ExCodecs.available_codecs/0`, or only this category with + `available_formats/0`. + + ## Quick start + + alias ExCodecs.Spatial.{Point, PointCloud} + + cloud = + PointCloud.new([ + Point.new(0.0, 0.0, 0.0, color: {255, 0, 0}), + Point.new(1.0, 1.0, 1.0, color: {0, 255, 0}) + ]) + + {:ok, ply} = ExCodecs.Spatial.encode(cloud, format: :ply) + {:ok, decoded} = ExCodecs.Spatial.decode(ply, format: :ply) + + ## Streaming note + + `stream_decode` / `stream_encode` currently materialize full payloads, then + enumerate. Prefer `source: :file` when the argument is a path, or + `source: :binary` for payloads. See `docs/spatial_formats.md` for `:auto` + path heuristics and wire-format layouts. + """ + + alias ExCodecs.{CodecRegistry, Error} + alias ExCodecs.Spatial.{GaussianCloud, PointCloud} + @formats [:ply, :spatial_binary, :gsplat] + + @doc """ + Lists the spatial formats accepted by this module. + + ## Arguments + + This function takes no arguments. + + ## Returns + + The list `[:ply, :spatial_binary, :gsplat]`, where `:ply` supports point and + Gaussian clouds, `:spatial_binary` supports point clouds, and `:gsplat` + supports Gaussian clouds. + + ## Raises / exceptions + + This function does not raise. + + ## Examples + + iex> ExCodecs.Spatial.available_formats() + [:ply, :spatial_binary, :gsplat] + """ + @spec available_formats() :: [atom()] + def available_formats do + available = CodecRegistry.available_codecs(:spatial) + preferred = Enum.filter(@formats, &(&1 in available)) + preferred ++ Enum.reject(available, &(&1 in @formats)) + end + + @doc """ + Tests whether an atom names a supported spatial format. + + ## Arguments + + * `format` (`atom()`) — the candidate format name. + + ## Returns + + `true` for `:ply`, `:spatial_binary`, or `:gsplat`; otherwise `false`. + + ## Raises / exceptions + + Raises `FunctionClauseError` when `format` is not an atom because the public + function is guarded with `is_atom/1`. + + ## Examples + + iex> ExCodecs.Spatial.supports?(:ply) + true + + iex> ExCodecs.Spatial.supports?(:sog) + false + """ + @spec supports?(atom()) :: boolean() + def supports?(format) when is_atom(format) do + case CodecRegistry.codec_info(format) do + {:ok, %{category: :spatial, interface: :spatial, module: module}} -> module != nil + _ -> false + end + end + + @doc """ + Encodes a point cloud or Gaussian cloud in a selected spatial format. + + ## Arguments + + * `data` (`PointCloud.t() | GaussianCloud.t()`) — a point cloud for + `:ply` or `:spatial_binary`, or a Gaussian cloud for `:ply` or `:gsplat`. + * `opts` (`keyword()`) — options passed to the selected codec: + * `:format` — spatial container: `:ply` (default), `:spatial_binary`, or + `:gsplat`. This key is removed before codec dispatch. + * for PLY, `:ply_format` selects `:ascii`, `:binary`, `:binary_le`, or + `:binary_be`; `:comments` overrides metadata comments. + * the binary and GSPL codecs currently ignore remaining options. + + `%PointCloud{}` contains a list of `%Point{}` values; `%GaussianCloud{}` + contains a list of `%Gaussian{}` values. See the selected codec for the + fields represented on the wire. + + ## Returns + + * `{:ok, payload}` where `payload` is the encoded `binary()`. + * `{:error, %ExCodecs.Error{reason: :invalid_data}}` when `data` is not a + supported cloud, the cloud type does not match the format, a PLY value + cannot be encoded, or a malformed struct causes PLY encoding to fail. + * `{:error, %ExCodecs.Error{reason: :unsupported_codec}}` when `:format` + is not one of the supported spatial format atoms. + + ## Raises / exceptions + + Validation errors above are returned. `Keyword.pop/3` raises + `FunctionClauseError` when `opts` is not a proper keyword list. EXCP and GSPL + encoding can also raise exceptions such as `MatchError`, `FunctionClauseError`, + or `ArgumentError` when callers manually construct malformed cloud/member + structs whose tuple shapes or numeric values violate the documented struct + contracts. PLY catches encoding exceptions and returns `:invalid_data`. + + ## Examples + + iex> alias ExCodecs.Spatial.{Point, PointCloud} + iex> cloud = PointCloud.new([Point.new(1.0, 2.0, 3.0)]) + iex> {:ok, bin} = ExCodecs.Spatial.encode(cloud, format: :ply) + iex> is_binary(bin) + true + """ + @spec encode(PointCloud.t() | GaussianCloud.t(), keyword()) :: + {:ok, binary()} | {:error, Error.t()} + def encode(data, opts \\ []) + + def encode(%PointCloud{} = data, opts) do + {format, codec_opts} = Keyword.pop(opts, :format, :ply) + + if format == :gsplat do + {:error, + Error.new(:invalid_data, + codec: :gsplat, + message: "GSPLAT format requires a GaussianCloud" + )} + else + with {:ok, module} <- spatial_codec(format) do + module.encode(data, codec_opts) + end + end + end + + def encode(%GaussianCloud{} = data, opts) do + {format, codec_opts} = Keyword.pop(opts, :format, :ply) + + if format == :spatial_binary do + {:error, + Error.new(:invalid_data, + codec: :spatial_binary, + message: "spatial_binary format requires a PointCloud" + )} + else + with {:ok, module} <- spatial_codec(format) do + module.encode(data, codec_opts) + end + end + end + + def encode(_, _) do + {:error, + Error.new(:invalid_data, + message: "Spatial encode expects a PointCloud or GaussianCloud" + )} + end + + @doc """ + Decodes a spatial payload into a point cloud or Gaussian cloud. + + ## Arguments + + * `data` (`binary()`) — a complete encoded payload. + * `opts` (`keyword()`) — decode options: + * `:format` — `:ply` (default), `:spatial_binary`, or `:gsplat`. + * `:as` — PLY interpretation: `:auto` (default), `:point_cloud`, or + `:gaussian_cloud`. In `:auto`, Gaussian property names select a + `%GaussianCloud{}`; other vertex schemas select a `%PointCloud{}`. + + ## Returns + + * `{:ok, %PointCloud{}}` for EXCP or point-oriented PLY. + * `{:ok, %GaussianCloud{}}` for GSPL or Gaussian-oriented PLY. + * `{:error, %ExCodecs.Error{reason: :invalid_data}}` for a non-binary + input, bad magic/version, malformed or unsupported PLY header/property, + missing/truncated records, or otherwise invalid payload. + * `{:error, %ExCodecs.Error{reason: :unsupported_codec}}` for an unknown + `:format`. + + ## Raises / exceptions + + Payload validation failures are returned. `Keyword.pop/3` and codec option + access raise `FunctionClauseError` when `opts` is not a proper keyword list. + For PLY, an unsupported `:as` value has no matching interpretation clause + and raises `CaseClauseError`; malformed ASCII rows can also surface + constructor/shape exceptions instead of an error tuple. + + ## Examples + + iex> alias ExCodecs.Spatial.{Point, PointCloud} + iex> {:ok, bin} = ExCodecs.Spatial.encode(PointCloud.new([Point.new(0.0, 0.0, 1.0)]), format: :ply) + iex> {:ok, %PointCloud{points: points}} = ExCodecs.Spatial.decode(bin, format: :ply) + iex> length(points) + 1 + """ + @spec decode(binary(), keyword()) :: + {:ok, PointCloud.t() | GaussianCloud.t()} | {:error, Error.t()} + def decode(data, opts \\ []) + + def decode(data, opts) when is_binary(data) do + {format, codec_opts} = Keyword.pop(opts, :format, :ply) + + with {:ok, module} <- spatial_codec(format) do + module.decode(data, codec_opts) + end + end + + def decode(_, _) do + {:error, Error.new(:invalid_data, message: "Spatial decode expects a binary")} + end + + defp spatial_codec(format) do + case CodecRegistry.lookup(format) do + {:ok, {module, :spatial, %{interface: :spatial}}} when module != nil -> + {:ok, module} + + {:ok, {nil, :spatial, %{interface: :spatial}}} -> + {:error, Error.new(:codec_unavailable, codec: format)} + + _ -> + {:error, Error.new(:unsupported_codec, codec: format)} + end + end + + @doc """ + Returns an enumerable over points or Gaussians decoded from a path or binary. + + Despite the name, decoding currently materializes the complete payload and + decoded cloud before yielding elements. + + ## Arguments + + * `source` (`Path.t() | binary()`) — a filesystem path or encoded payload. + Since paths are binaries in Elixir, use `source: :file` to force path + interpretation or `source: :binary` to force payload interpretation. + * `opts` (`keyword()`) — requires `:format` (`:ply`, + `:spatial_binary`, or `:gsplat`). `:source` may be `:auto` (default), + `:file`, or `:binary`; remaining options go to the selected decoder. + + ## Returns + + An `Enumerable.t()` yielding `%Point{}` or `%Gaussian{}`. On failure it + yields exactly one `{:error, %ExCodecs.Error{}}` element: + + * `reason: :invalid_options` when `:format` is missing. + * `reason: :unsupported_codec` for an unknown format. + * `reason: :io_error` when a forced or auto-detected file cannot be read. + * `reason: :invalid_data` for malformed, unsupported, or truncated data. + + ## Raises / exceptions + + Missing `:format` is represented by an error element. Keyword operations + raise `FunctionClauseError` for a non-keyword `opts`. PLY only accepts a + binary/path source and raises `FunctionClauseError` otherwise. Invalid + `:source` values can raise `CaseClauseError`. Exceptions documented by the + selected decoder can occur while the returned stream is enumerated. + + ## Examples + + iex> alias ExCodecs.Spatial.{Point, PointCloud} + iex> {:ok, bin} = ExCodecs.Spatial.encode(PointCloud.new([Point.new(1.0, 0.0, 0.0)]), format: :ply) + iex> list = ExCodecs.Spatial.stream_decode(bin, format: :ply) |> Enum.to_list() + iex> match?([%Point{}], list) + true + """ + @spec stream_decode(Path.t() | binary(), keyword()) :: Enumerable.t() + def stream_decode(source, opts \\ []) do + ExCodecs.Spatial.Stream.decode(source, opts) + end + + @doc """ + Encodes an enumerable of points or Gaussians to one spatial payload. + + The enumerable is fully collected so the codec can write the element count. + A non-empty enumerable is classified by its first element and must contain + valid values of that same struct type. An empty enumerable becomes an empty + point cloud for PLY/EXCP or an empty Gaussian cloud for GSPL. + + ## Arguments + + * `enumerable` (`Enumerable.t()`) — `%Point{}` or `%Gaussian{}` elements. + * `opts` (`keyword()`) — requires `:format` (`:ply`, + `:spatial_binary`, or `:gsplat`); remaining options are the same as + `encode/2`. + + ## Returns + + * `{:ok, payload}` where `payload` is a `binary()`. + * `{:error, %ExCodecs.Error{reason: :invalid_options}}` when `:format` is + missing. + * `{:error, %ExCodecs.Error{reason: :unsupported_codec}}` for an unknown + format. + * `{:error, %ExCodecs.Error{reason: :invalid_data}}` when the first element + is not a point/Gaussian, is an error tuple, the primitive type does not + match the format, or codec encoding fails. + + ## Raises / exceptions + + Missing `:format` is returned as `:invalid_options`. `Enum.to_list/1` may + raise `Protocol.UndefinedError` for a non-enumerable or propagate an + exception raised by the enumerable. Keyword operations raise + `FunctionClauseError` for invalid `opts`; malformed member structs may raise + the codec exceptions described by `encode/2`. + + ## Examples + + iex> alias ExCodecs.Spatial.Point + iex> {:ok, bin} = ExCodecs.Spatial.stream_encode([Point.new(0.0, 0.0, 0.0)], format: :spatial_binary) + iex> is_binary(bin) + true + """ + @spec stream_encode(Enumerable.t(), keyword()) :: {:ok, binary()} | {:error, Error.t()} + def stream_encode(enumerable, opts \\ []) do + ExCodecs.Spatial.Stream.encode(enumerable, opts) + end +end diff --git a/lib/ex_codecs/spatial/bounds.ex b/lib/ex_codecs/spatial/bounds.ex new file mode 100644 index 0000000..13217af --- /dev/null +++ b/lib/ex_codecs/spatial/bounds.ex @@ -0,0 +1,215 @@ +defmodule ExCodecs.Spatial.Bounds do + @moduledoc """ + An axis-aligned bounding box. + + ## Fields + + * `:min_x`, `:min_y`, `:min_z` — `float()` lower Cartesian limits. + * `:max_x`, `:max_y`, `:max_z` — `float()` upper Cartesian limits. + + All six fields are enforced when constructing a literal struct. Their raw + `defstruct` defaults are `nil`, but `new/2` stores floats. Units are + application-defined and must match the bounded coordinates. Bounds are + inclusive, and this module does not require each minimum to be no greater + than its corresponding maximum. + + ## Example + + iex> alias ExCodecs.Spatial.Bounds + iex> bounds = Bounds.new({-1, 0, 2}, {4, 5, 8}) + iex> {Bounds.center(bounds), Bounds.contains?(bounds, {0, 1, 3})} + {{1.5, 2.5, 5.0}, true} + """ + + @typedoc """ + An inclusive axis-aligned bounding box. See the module documentation for + every field, defaults, units and conventions, and a construction example. + """ + @type t :: %__MODULE__{ + min_x: float(), + min_y: float(), + min_z: float(), + max_x: float(), + max_y: float(), + max_z: float() + } + + @enforce_keys [:min_x, :min_y, :min_z, :max_x, :max_y, :max_z] + defstruct [:min_x, :min_y, :min_z, :max_x, :max_y, :max_z] + + @doc """ + Builds bounds from min and max corners. + + ## Arguments + + * `min` — numeric `{min_x, min_y, min_z}` lower corner in + application-defined coordinate units. + * `max` — numeric `{max_x, max_y, max_z}` upper corner in the same units. + + ## Returns + + `%Bounds{}` + + ## Raises + + * `FunctionClauseError` if either argument is not a 3-tuple. + * `ArithmeticError` if any component is not numeric. + + ## Examples + + iex> b = ExCodecs.Spatial.Bounds.new({0, 0, 0}, {1, 2, 3}) + iex> b.max_z + 3.0 + """ + @spec new( + {number(), number(), number()}, + {number(), number(), number()} + ) :: t() + def new({min_x, min_y, min_z}, {max_x, max_y, max_z}) do + %__MODULE__{ + min_x: min_x * 1.0, + min_y: min_y * 1.0, + min_z: min_z * 1.0, + max_x: max_x * 1.0, + max_y: max_y * 1.0, + max_z: max_z * 1.0 + } + end + + @doc """ + Computes bounds from points or `{x,y,z}` tuples. + + ## Arguments + + * `points` — `Enumerable.t()` of maps/structs with numeric `:x`, `:y`, + and `:z` fields, or numeric `{x, y, z}` tuples. + + ## Returns + + `%Bounds{}` or `nil` if empty + + ## Raises + + * `Protocol.UndefinedError` if `points` is not enumerable. + * `FunctionClauseError` if an element is not a supported point-like value. + * `ArithmeticError` if a map/struct coordinate is not numeric. + + ## Examples + + iex> ExCodecs.Spatial.Bounds.from_points([]) + nil + iex> b = ExCodecs.Spatial.Bounds.from_points([{0, 0, 0}, {1, 1, 1}]) + iex> b.max_x + 1.0 + """ + @spec from_points(Enumerable.t()) :: t() | nil + def from_points(points) do + Enum.reduce(points, nil, fn point, acc -> + {x, y, z} = coords(point) + + case acc do + nil -> + new({x, y, z}, {x, y, z}) + + %__MODULE__{} = b -> + %__MODULE__{ + min_x: min(b.min_x, x), + min_y: min(b.min_y, y), + min_z: min(b.min_z, z), + max_x: max(b.max_x, x), + max_y: max(b.max_y, y), + max_z: max(b.max_z, z) + } + end + end) + end + + @doc """ + Center point of the box. + + ## Arguments + + * `bounds` — `%Bounds{}` + + ## Returns + + `{cx, cy, cz}` floats + + ## Raises + + * `FunctionClauseError` if `bounds` is not a `%Bounds{}`. + * `ArithmeticError` if a manually constructed bounds has non-numeric fields. + + ## Examples + + iex> ExCodecs.Spatial.Bounds.center(ExCodecs.Spatial.Bounds.new({0, 0, 0}, {2, 2, 2})) + {1.0, 1.0, 1.0} + """ + @spec center(t()) :: {float(), float(), float()} + def center(%__MODULE__{} = b) do + {(b.min_x + b.max_x) / 2.0, (b.min_y + b.max_y) / 2.0, (b.min_z + b.max_z) / 2.0} + end + + @doc """ + Extents `{dx, dy, dz}`. + + ## Arguments + + * `bounds` — `%Bounds{}` + + ## Returns + + `{float(), float(), float()}` + + ## Raises + + * `FunctionClauseError` if `bounds` is not a `%Bounds{}`. + * `ArithmeticError` if a manually constructed bounds has non-numeric fields. + + ## Examples + + iex> ExCodecs.Spatial.Bounds.size(ExCodecs.Spatial.Bounds.new({0, 0, 0}, {1, 2, 3})) + {1.0, 2.0, 3.0} + """ + @spec size(t()) :: {float(), float(), float()} + def size(%__MODULE__{} = b) do + {b.max_x - b.min_x, b.max_y - b.min_y, b.max_z - b.min_z} + end + + @doc """ + Whether `{x,y,z}` lies inside or on the box. + + ## Arguments + + * `bounds` — `%Bounds{}` + * `point` — `{number(), number(), number()}` in the bounds' coordinate + units. + + ## Returns + + boolean + + ## Raises + + * `FunctionClauseError` unless the arguments are a `%Bounds{}` and a + 3-tuple. Elixir term ordering is otherwise used for non-numeric values + supplied outside the documented types. + + ## Examples + + iex> b = ExCodecs.Spatial.Bounds.new({0, 0, 0}, {1, 1, 1}) + iex> ExCodecs.Spatial.Bounds.contains?(b, {0.5, 0.5, 0.5}) + true + """ + @spec contains?(t(), {number(), number(), number()}) :: boolean() + def contains?(%__MODULE__{} = b, {x, y, z}) do + x >= b.min_x and x <= b.max_x and + y >= b.min_y and y <= b.max_y and + z >= b.min_z and z <= b.max_z + end + + defp coords(%{x: x, y: y, z: z}), do: {x * 1.0, y * 1.0, z * 1.0} + + defp coords({x, y, z}) when is_number(x) and is_number(y) and is_number(z), + do: {x * 1.0, y * 1.0, z * 1.0} +end diff --git a/lib/ex_codecs/spatial/codec/binary.ex b/lib/ex_codecs/spatial/codec/binary.ex new file mode 100644 index 0000000..5fabef6 --- /dev/null +++ b/lib/ex_codecs/spatial/codec/binary.ex @@ -0,0 +1,311 @@ +defmodule ExCodecs.Spatial.Codec.Binary do + @moduledoc """ + Compact little-endian binary format for point clouds. + + ## Layout + + magic: "EXCP" (4 bytes) + version: u16 LE = 1 + flags: u16 LE (bit0=color, bit1=alpha, bit2=normal) + count: u64 LE + records: count × record + + Each record always has `x,y,z` as `f32`. Optional `rgb`/`rgba` as `u8` + and normals as `f32` follow according to flags. + + Generic attributes are not stored in this format — use PLY for that. + """ + + alias ExCodecs.Error + alias ExCodecs.Spatial.{Metadata, Point, PointCloud} + + @magic "EXCP" + @version 1 + + @flag_color 0b001 + @flag_alpha 0b010 + @flag_normal 0b100 + + @doc """ + Encodes a point cloud in the EXCP version 1 binary format. + + The 16-byte header is `"EXCP"`, version `u16` little-endian, global flags + `u16` little-endian, and point count `u64` little-endian. Each record starts + with three little-endian `f32` coordinates. Global flag bits then add RGB + (3 bytes), RGBA (4 bytes; alpha takes precedence), and/or three little-endian + `f32` normal components to every record. + + Flags are selected from the whole cloud. Consequently, points missing an + optional field are zero-filled (missing alpha is 255). Point attributes, + cloud bounds, transform, and metadata are not represented. + + ## Arguments + + * `data` (`PointCloud.t()`) — a cloud whose `points` are valid `%Point{}` + structs with numeric coordinates, optional `{r, g, b}` or + `{r, g, b, a}` color, and optional `{nx, ny, nz}` normal. + * `opts` (`keyword()`) — reserved and currently ignored. + + ## Returns + + * `{:ok, payload}` where `payload` is an EXCP `binary()`. + * `{:error, %ExCodecs.Error{reason: :invalid_data, + codec: :spatial_binary}}` when `data` is not a `%PointCloud{}`. + + ## Raises / exceptions + + A wrong top-level type is returned as `:invalid_data`. Because `opts` is + ignored, even a non-keyword value does not raise. Manually malformed cloud or + point structs can raise `FunctionClauseError`, `MatchError`, `ArgumentError`, + or a bitstring construction exception when a point, tuple shape, channel + range, or numeric field violates the struct contract. + + ## Examples + + iex> alias ExCodecs.Spatial.{Point, PointCloud} + iex> cloud = PointCloud.new([Point.new(1, 2, 3, color: {10, 20, 30})]) + iex> {:ok, <<"EXCP", 1::little-16, flags::little-16, 1::little-64, _::binary>>} = + ...> ExCodecs.Spatial.Codec.Binary.encode(cloud) + iex> Bitwise.band(flags, 0b001) + 1 + """ + @spec encode(PointCloud.t(), keyword()) :: {:ok, binary()} | {:error, Error.t()} + def encode(data, opts \\ []) + + def encode(%PointCloud{points: points}, _opts) do + has_color? = Enum.any?(points, &Point.colored?/1) + has_alpha? = Enum.any?(points, fn p -> match?({_, _, _, _}, p.color) end) + has_normal? = Enum.any?(points, &Point.has_normal?/1) + + flags = + 0 + |> then(fn f -> if has_color?, do: Bitwise.bor(f, @flag_color), else: f end) + |> then(fn f -> if has_alpha?, do: Bitwise.bor(f, @flag_alpha), else: f end) + |> then(fn f -> if has_normal?, do: Bitwise.bor(f, @flag_normal), else: f end) + + header = + <<@magic::binary, @version::little-unsigned-16, flags::little-unsigned-16, + length(points)::little-unsigned-64>> + + body = + IO.iodata_to_binary( + Enum.map(points, fn p -> + encode_point(p, flags) + end) + ) + + {:ok, header <> body} + end + + def encode(_, _) do + {:error, + Error.new(:invalid_data, + codec: :spatial_binary, + message: "Binary encode expects a PointCloud" + )} + end + + @doc """ + Decodes an EXCP version 1 payload into a point cloud. + + The decoder reads the global color/alpha/normal flags and the declared number + of fixed-shape records described by `encode/2`. Trailing bytes are currently + ignored. Decoded metadata contains `%{"format" => "excp", "version" => 1}`; + attributes and other cloud-level fields use their constructor defaults. + + ## Arguments + + * `data` (`binary()`) — a complete EXCP payload. + * `opts` (`keyword()`) — reserved and currently ignored. + + ## Returns + + * `{:ok, %PointCloud{}}` + * `{:error, %ExCodecs.Error{reason: :invalid_data, + codec: :spatial_binary}}` when the magic/header is invalid or too short, + the version is not 1, or a declared coordinate, RGB/RGBA, or normal field + is truncated. + + ## Raises / exceptions + + Corrupt/truncated payloads covered above are returned, and `opts` is ignored. + This function has a catch-all data clause, so non-binary input also returns + `:invalid_data`; it does not intentionally raise for external payloads. + + ## Examples + + iex> alias ExCodecs.Spatial.{Point, PointCloud} + iex> {:ok, bin} = ExCodecs.Spatial.Codec.Binary.encode(PointCloud.new([Point.new(1.0, 2.0, 3.0)])) + iex> {:ok, %PointCloud{points: [p]}} = ExCodecs.Spatial.Codec.Binary.decode(bin) + iex> p.x + 1.0 + """ + @spec decode(binary(), keyword()) :: {:ok, PointCloud.t()} | {:error, Error.t()} + def decode(data, opts \\ []) + + def decode( + <<@magic::binary, version::little-unsigned-16, flags::little-unsigned-16, + count::little-unsigned-64, rest::binary>>, + _opts + ) do + if version != @version do + {:error, + Error.new(:invalid_data, + codec: :spatial_binary, + message: "Unsupported binary point format version #{version}" + )} + else + with {:ok, points, _} <- decode_points(rest, count, flags) do + meta = Metadata.new(entries: %{"format" => "excp", "version" => version}) + {:ok, PointCloud.new(points, metadata: meta)} + end + end + end + + def decode(_, _) do + {:error, + Error.new(:invalid_data, + codec: :spatial_binary, + message: "Invalid ExCodecs binary point cloud" + )} + end + + @doc """ + Returns an enumerable over points in an EXCP payload. + + ## Arguments + + * `data` (`binary()`) — a complete EXCP payload. + * `opts` (`keyword()`) — reserved and currently ignored. + + ## Returns + + An `Enumerable.t()` that yields decoded `%Point{}` values after the complete + cloud has been materialized. If `decode/2` fails, it yields exactly one + `{:error, %ExCodecs.Error{reason: :invalid_data, + codec: :spatial_binary}}` element. + + ## Raises / exceptions + + Raises `FunctionClauseError` when `data` is not a binary because this public + function is guarded. `opts` is ignored. Binary validation failures are + delayed as the single error element rather than raised. + + ## Examples + + iex> alias ExCodecs.Spatial.{Point, PointCloud} + iex> {:ok, bin} = ExCodecs.Spatial.Codec.Binary.encode(PointCloud.new([Point.new(0, 0, 0)])) + iex> [%Point{x: 0.0, y: 0.0, z: 0.0}] = + ...> ExCodecs.Spatial.Codec.Binary.stream_decode(bin) |> Enum.to_list() + """ + @spec stream_decode(binary(), keyword()) :: Enumerable.t() + def stream_decode(data, opts \\ []) when is_binary(data) do + case decode(data, opts) do + {:ok, %PointCloud{points: points}} -> + Stream.map(points, & &1) + + {:error, error} -> + Stream.resource( + fn -> {:error, error} end, + fn + {:error, e} -> {[{:error, e}], :done} + :done -> {:halt, :done} + end, + fn _ -> :ok end + ) + end + end + + defp encode_point(%Point{} = p, flags) do + xyz = <> + + color = + cond do + Bitwise.band(flags, @flag_alpha) != 0 -> + {r, g, b, a} = normalize_rgba(p.color) + <> + + Bitwise.band(flags, @flag_color) != 0 -> + {r, g, b} = normalize_rgb(p.color) + <> + + true -> + <<>> + end + + normal = + if Bitwise.band(flags, @flag_normal) != 0 do + {nx, ny, nz} = p.normal || {0.0, 0.0, 0.0} + <> + else + <<>> + end + + xyz <> color <> normal + end + + defp normalize_rgb(nil), do: {0, 0, 0} + defp normalize_rgb({r, g, b}), do: {trunc(r), trunc(g), trunc(b)} + defp normalize_rgb({r, g, b, _a}), do: {trunc(r), trunc(g), trunc(b)} + + defp normalize_rgba(nil), do: {0, 0, 0, 255} + defp normalize_rgba({r, g, b}), do: {trunc(r), trunc(g), trunc(b), 255} + defp normalize_rgba({r, g, b, a}), do: {trunc(r), trunc(g), trunc(b), trunc(a)} + + defp decode_points(bin, 0, _flags), do: {:ok, [], bin} + + defp decode_points(bin, count, flags) do + Enum.reduce_while(1..count, {:ok, [], bin}, fn _, {:ok, acc, rest} -> + case decode_point(rest, flags) do + {:ok, point, next} -> {:cont, {:ok, [point | acc], next}} + {:error, _} = err -> {:halt, err} + end + end) + |> case do + {:ok, points, rest} -> {:ok, Enum.reverse(points), rest} + other -> other + end + end + + defp decode_point( + <>, + flags + ) do + with {:ok, color, rest} <- take_color(rest, flags), + {:ok, normal, rest} <- take_normal(rest, flags) do + {:ok, Point.new(x, y, z, color: color, normal: normal), rest} + end + end + + defp decode_point(_, _), + do: + {:error, Error.new(:invalid_data, codec: :spatial_binary, message: "Truncated point record")} + + defp take_color(bin, flags) when Bitwise.band(flags, @flag_alpha) != 0 do + case bin do + <> -> {:ok, {r, g, b, a}, rest} + _ -> {:error, Error.new(:invalid_data, codec: :spatial_binary, message: "Truncated RGBA")} + end + end + + defp take_color(bin, flags) when Bitwise.band(flags, @flag_color) != 0 do + case bin do + <> -> {:ok, {r, g, b}, rest} + _ -> {:error, Error.new(:invalid_data, codec: :spatial_binary, message: "Truncated RGB")} + end + end + + defp take_color(bin, _), do: {:ok, nil, bin} + + defp take_normal(bin, flags) when Bitwise.band(flags, @flag_normal) != 0 do + case bin do + <> -> + {:ok, {nx, ny, nz}, rest} + + _ -> + {:error, Error.new(:invalid_data, codec: :spatial_binary, message: "Truncated normal")} + end + end + + defp take_normal(bin, _), do: {:ok, nil, bin} +end diff --git a/lib/ex_codecs/spatial/codec/gsplat.ex b/lib/ex_codecs/spatial/codec/gsplat.ex new file mode 100644 index 0000000..4d43cee --- /dev/null +++ b/lib/ex_codecs/spatial/codec/gsplat.ex @@ -0,0 +1,311 @@ +defmodule ExCodecs.Spatial.Codec.Gsplat do + @moduledoc """ + Simple little-endian binary format for Gaussian splat clouds. + + ## Layout + + magic: "GSPL" (4 bytes) + version: u16 LE = 1 + flags: u16 LE (bit0 = has SH rest coeffs count in header) + count: u64 LE + sh_rest: u16 LE (number of f_rest floats per Gaussian; 0 if none) + records: count × record + + Each record: + + position: 3 × f32 + color: 3 × f32 (f_dc) + opacity: f32 + scale: 3 × f32 + rotation: 4 × f32 (w, x, y, z) + sh_rest: sh_rest × f32 (optional) + """ + + alias ExCodecs.Error + alias ExCodecs.Spatial.{Gaussian, GaussianCloud, Metadata} + + @magic "GSPL" + @version 1 + + @doc """ + Encodes a Gaussian cloud in the GSPL version 1 binary format. + + The 18-byte header contains `"GSPL"`, version `u16` little-endian, flags + `u16` little-endian, Gaussian count `u64` little-endian, and a shared + spherical-harmonic-rest count `u16` little-endian. Each record contains 14 + little-endian `f32` values: position XYZ, DC color RGB, opacity, scale XYZ, + and quaternion rotation `(w, x, y, z)`, followed by the shared number of + little-endian `f32` SH-rest values. + + The shared SH count is the longest flattened rest coefficient list in the + cloud; shorter lists are padded with zero. The DC coefficient is represented + by `Gaussian.color`. Gaussian metadata and cloud-level metadata are not + stored. + + ## Arguments + + * `data` (`GaussianCloud.t()`) — a cloud of valid `%Gaussian{}` structs. + Each Gaussian has position/color/scale 3-tuples, a rotation 4-tuple, + numeric opacity, and optional SH data shaped as `[dc | rest]` (nested + rest lists are flattened). + * `opts` (`keyword()`) — reserved and currently ignored. + + ## Returns + + * `{:ok, payload}` where `payload` is a GSPL `binary()`. + * `{:error, %ExCodecs.Error{reason: :invalid_data, codec: :gsplat}}` when + `data` is not a `%GaussianCloud{}`. + + ## Raises / exceptions + + A wrong top-level type is returned as `:invalid_data`, and `opts` is ignored. + Manually malformed cloud/Gaussian structs can raise `FunctionClauseError`, + `MatchError`, `ArgumentError`, `ArithmeticError`, or a bitstring construction + exception for invalid SH shapes, tuple shapes, counts, or non-numeric values. + + ## Examples + + iex> alias ExCodecs.Spatial.{Gaussian, GaussianCloud} + iex> cloud = GaussianCloud.new([Gaussian.new({0, 0, 0}, opacity: 0.75)]) + iex> {:ok, <<"GSPL", 1::little-16, _flags::little-16, 1::little-64, + ...> 0::little-16, _record::binary>>} = + ...> ExCodecs.Spatial.Codec.Gsplat.encode(cloud) + iex> true + true + """ + @spec encode(GaussianCloud.t(), keyword()) :: {:ok, binary()} | {:error, Error.t()} + def encode(data, opts \\ []) + + def encode(%GaussianCloud{gaussians: gaussians}, _opts) do + sh_rest = max_sh_rest(gaussians) + flags = if sh_rest > 0, do: 1, else: 0 + + header = + <<@magic::binary, @version::little-unsigned-16, flags::little-unsigned-16, + length(gaussians)::little-unsigned-64, sh_rest::little-unsigned-16>> + + body = + IO.iodata_to_binary(Enum.map(gaussians, fn g -> encode_gaussian(g, sh_rest) end)) + + {:ok, header <> body} + end + + def encode(_, _) do + {:error, + Error.new(:invalid_data, codec: :gsplat, message: "GSPLAT encode expects a GaussianCloud")} + end + + @doc """ + Decodes a GSPL version 1 payload into a Gaussian cloud. + + The decoder reads the header and declared record count described by + `encode/2`. When the shared SH-rest count is nonzero, each decoded + Gaussian's `sh` is `[[r, g, b] | Enum.chunk_every(rest, 3)]`; otherwise it + is `nil`. Trailing bytes and unknown flag bits are currently ignored. + Decoded cloud metadata contains `%{"format" => "gsplat", "version" => 1}`. + + ## Arguments + + * `data` (`binary()`) — a complete GSPL payload. + * `opts` (`keyword()`) — reserved and currently ignored. + + ## Returns + + * `{:ok, %GaussianCloud{}}` + * `{:error, %ExCodecs.Error{reason: :invalid_data, codec: :gsplat}}` when + the magic/header is invalid or too short, the version is not 1, a + declared 14-float record is truncated, or its declared SH-rest values + are truncated. + + ## Raises / exceptions + + The external payload failures above are returned. `opts` is ignored, and + non-binary input is handled by the catch-all clause as `:invalid_data`; this + function does not intentionally raise for external payloads. + + ## Examples + + iex> alias ExCodecs.Spatial.{Gaussian, GaussianCloud} + iex> {:ok, bin} = ExCodecs.Spatial.Codec.Gsplat.encode(GaussianCloud.new([Gaussian.new({1.0, 0.0, 0.0})])) + iex> {:ok, %GaussianCloud{gaussians: [g]}} = ExCodecs.Spatial.Codec.Gsplat.decode(bin) + iex> elem(g.position, 0) + 1.0 + """ + @spec decode(binary(), keyword()) :: {:ok, GaussianCloud.t()} | {:error, Error.t()} + def decode(data, opts \\ []) + + def decode( + <<@magic::binary, version::little-unsigned-16, _flags::little-unsigned-16, + count::little-unsigned-64, sh_rest::little-unsigned-16, rest::binary>>, + _opts + ) do + if version != @version do + {:error, + Error.new(:invalid_data, + codec: :gsplat, + message: "Unsupported GSPLAT version #{version}" + )} + else + with {:ok, gaussians, _} <- decode_gaussians(rest, count, sh_rest) do + meta = Metadata.new(entries: %{"format" => "gsplat", "version" => version}) + {:ok, GaussianCloud.new(gaussians, metadata: meta)} + end + end + end + + def decode(_, _) do + {:error, Error.new(:invalid_data, codec: :gsplat, message: "Invalid GSPLAT binary")} + end + + @doc """ + Returns an enumerable over Gaussians in a GSPL payload. + + ## Arguments + + * `data` (`binary()`) — a complete GSPL payload. + * `opts` (`keyword()`) — reserved and currently ignored. + + ## Returns + + An `Enumerable.t()` that yields decoded `%Gaussian{}` structs after the + complete cloud has been materialized. If `decode/2` fails, it yields exactly + one `{:error, %ExCodecs.Error{reason: :invalid_data, codec: :gsplat}}` + element. + + ## Raises / exceptions + + Raises `FunctionClauseError` when `data` is not a binary because this public + function is guarded. `opts` is ignored. Binary validation failures are + delayed as the single error element rather than raised. + + ## Examples + + iex> alias ExCodecs.Spatial.{Gaussian, GaussianCloud} + iex> {:ok, bin} = ExCodecs.Spatial.Codec.Gsplat.encode(GaussianCloud.new([Gaussian.new({0, 0, 0})])) + iex> [%Gaussian{opacity: 1.0}] = + ...> ExCodecs.Spatial.Codec.Gsplat.stream_decode(bin) |> Enum.to_list() + """ + @spec stream_decode(binary(), keyword()) :: Enumerable.t() + def stream_decode(data, opts \\ []) when is_binary(data) do + case decode(data, opts) do + {:ok, %GaussianCloud{gaussians: gs}} -> + Stream.map(gs, & &1) + + {:error, error} -> + Stream.resource( + fn -> {:error, error} end, + fn + {:error, e} -> {[{:error, e}], :done} + :done -> {:halt, :done} + end, + fn _ -> :ok end + ) + end + end + + defp max_sh_rest(gaussians) do + gaussians + |> Enum.map(fn + %{sh: nil} -> 0 + %{sh: [_dc | rest]} -> length(List.flatten(rest)) + %{sh: other} when is_list(other) -> max(length(List.flatten(other)) - 3, 0) + end) + |> Enum.max(fn -> 0 end) + end + + defp encode_gaussian(%Gaussian{} = g, sh_rest) do + {x, y, z} = g.position + {r, gc, b} = g.color + {sx, sy, sz} = g.scale + {rw, rx, ry, rz} = g.rotation + + base = + <> + + rest = + g.sh + |> sh_rest_values() + |> then(fn vals -> + vals + |> Stream.concat(Stream.cycle([0.0])) + |> Enum.take(sh_rest) + end) + |> Enum.map(fn v -> <> end) + |> IO.iodata_to_binary() + + base <> rest + end + + defp sh_rest_values(nil), do: [] + defp sh_rest_values([_dc | rest]), do: List.flatten(rest) + defp sh_rest_values(list) when is_list(list), do: list |> List.flatten() |> Enum.drop(3) + + defp decode_gaussians(bin, 0, _sh_rest), do: {:ok, [], bin} + + defp decode_gaussians(bin, count, sh_rest) do + Enum.reduce_while(1..count, {:ok, [], bin}, fn _, {:ok, acc, rest} -> + case decode_gaussian(rest, sh_rest) do + {:ok, g, next} -> {:cont, {:ok, [g | acc], next}} + {:error, _} = err -> {:halt, err} + end + end) + |> case do + {:ok, gs, rest} -> {:ok, Enum.reverse(gs), rest} + other -> other + end + end + + defp decode_gaussian( + <>, + sh_rest + ) do + case take_floats(rest, sh_rest) do + {:ok, sh_vals, next} -> + sh = + if sh_rest == 0 do + nil + else + [[r, gc, b] | Enum.chunk_every(sh_vals, 3)] + end + + g = + Gaussian.new({x, y, z}, + color: {r, gc, b}, + opacity: opacity, + scale: {sx, sy, sz}, + rotation: {rw, rx, ry, rz}, + sh: sh + ) + + {:ok, g, next} + + {:error, _} = err -> + err + end + end + + defp decode_gaussian(_, _) do + {:error, Error.new(:invalid_data, codec: :gsplat, message: "Truncated Gaussian record")} + end + + defp take_floats(bin, 0), do: {:ok, [], bin} + + defp take_floats(bin, n) do + Enum.reduce_while(1..n, {:ok, [], bin}, fn _, {:ok, acc, rest} -> + case rest do + <> -> {:cont, {:ok, [v | acc], next}} + _ -> {:halt, {:error, Error.new(:invalid_data, codec: :gsplat, message: "Truncated SH")}} + end + end) + |> case do + {:ok, vals, rest} -> {:ok, Enum.reverse(vals), rest} + other -> other + end + end +end diff --git a/lib/ex_codecs/spatial/codec/ply.ex b/lib/ex_codecs/spatial/codec/ply.ex new file mode 100644 index 0000000..18c07ab --- /dev/null +++ b/lib/ex_codecs/spatial/codec/ply.ex @@ -0,0 +1,961 @@ +defmodule ExCodecs.Spatial.Codec.PLY do + @moduledoc """ + ASCII and binary PLY for point clouds and Gaussian splats. + + ## Options + + * `:format` / `:ply_format` — wire encoding: `:ascii` (default), + `:binary` / `:binary_le`, `:binary_be` (not Spatial's `format: :ply`) + * `:comments` — header comments + * `:as` — decode as `:auto` | `:point_cloud` | `:gaussian_cloud` + * `:source` — for `stream_decode/2`: `:auto` | `:file` | `:binary` + + Public functions: `encode/2`, `decode/2`, `stream_decode/2`. + """ + + alias ExCodecs.Error + alias ExCodecs.Spatial.{Gaussian, GaussianCloud, Metadata, Point, PointCloud} + + @typedoc """ + Wire encoding used for a PLY payload. + + `:ascii` writes textual scalar values. `:binary_le` and `:binary_be` write + packed scalar values in little- and big-endian order, respectively. + `:binary` is an encoding-option shorthand for `:binary_le`; decoded headers + report the explicit endian atom. + + For example, `encode(cloud, ply_format: :binary_be)` emits a + `format binary_big_endian 1.0` header. + """ + @type ply_format :: :ascii | :binary | :binary_le | :binary_be + + @doc """ + Encodes a point cloud or Gaussian cloud as a PLY 1.0 payload. + + Point vertices always contain `x`, `y`, and `z` as `float`. If any point has + color or a normal, the corresponding `uchar` RGB/RGBA or `float` normal + properties are emitted for every point; missing values are zero-filled + (alpha defaults to 255). The union of point attribute keys is emitted as + sorted `float` properties, with absent attributes set to `0.0`. + + Gaussian vertices contain position, `f_dc_0..2`, opacity, scale, and + quaternion `rot_0..3` (`w, x, y, z`) as `float`. If spherical-harmonic + coefficients are present, flattened `f_rest_N` properties are appended and + shorter rows are zero-filled. + + ## Arguments + + * `data` (`PointCloud.t() | GaussianCloud.t()`) — a cloud containing valid + `%Point{}` or `%Gaussian{}` structs. + * `opts` (`keyword()`) — supported options: + * `:ply_format` — preferred wire format: `:ascii` (default), `:binary` + (little-endian), `:binary_le`, or `:binary_be`. + * `:format` — legacy alias for `:ply_format` when calling this codec + directly. `:ply_format` takes precedence. + * `:comments` — enumerable of strings for PLY `comment` header lines; + defaults to `cloud.metadata.comments`. + + `:little` and `:big` are also normalized to little-/big-endian output. + Any unrecognized wire-format value currently falls back to `:ascii`. + + ## Returns + + * `{:ok, payload}` where `payload` is an ASCII or binary PLY `binary()`. + * `{:error, %ExCodecs.Error{reason: :invalid_data, codec: :ply}}` when + `data` is not a supported cloud or any exception occurs while deriving + properties, building the header, or encoding rows. + + ## Raises / exceptions + + The cloud clauses rescue encoding exceptions and return `:invalid_data`, + including invalid keyword options, malformed nested structs, non-string + comments, invalid scalar ranges/types, and bad attribute shapes. Unsupported + top-level data is also returned as `:invalid_data`; this function does not + intentionally raise. + + ## Examples + + iex> alias ExCodecs.Spatial.{Point, PointCloud} + iex> cloud = PointCloud.new([Point.new(0, 1, 2, color: {255, 64, 0})]) + iex> {:ok, ply} = ExCodecs.Spatial.Codec.PLY.encode(cloud, ply_format: :ascii) + iex> String.contains?(ply, "property uchar red") + true + """ + @spec encode(PointCloud.t() | GaussianCloud.t(), keyword()) :: + {:ok, binary()} | {:error, Error.t()} + def encode(data, opts \\ []) + + def encode(%PointCloud{} = cloud, opts) do + format = normalize_format(ply_format_opt(opts)) + comments = Keyword.get(opts, :comments, cloud.metadata.comments) + + with {:ok, props} <- point_properties(cloud.points) do + body = encode_points(cloud.points, props, format) + header = build_header(format, length(cloud.points), props, comments) + {:ok, header <> body} + end + rescue + e -> + {:error, + Error.new(:invalid_data, + codec: :ply, + message: "PLY encode failed: #{Exception.message(e)}" + )} + end + + def encode(%GaussianCloud{} = cloud, opts) do + format = normalize_format(ply_format_opt(opts)) + comments = Keyword.get(opts, :comments, cloud.metadata.comments) + props = gaussian_properties(cloud.gaussians) + body = encode_gaussians(cloud.gaussians, props, format) + header = build_header(format, length(cloud.gaussians), props, comments) + {:ok, header <> body} + rescue + e -> + {:error, + Error.new(:invalid_data, + codec: :ply, + message: "PLY encode failed: #{Exception.message(e)}" + )} + end + + def encode(_data, _opts) do + {:error, + Error.new(:invalid_data, + codec: :ply, + message: "PLY encode expects a PointCloud or GaussianCloud" + )} + end + + @doc """ + Decodes a complete PLY 1.0 payload into a point or Gaussian cloud. + + ASCII and binary little-/big-endian scalar vertex properties are supported: + `char`/`int8`, `uchar`/`uint8`, `short`/`int16`, `ushort`/`uint16`, + `int`/`int32`, `uint`/`uint32`, `float`/`float32`, and + `double`/`float64`. List properties and non-vertex payloads are unsupported. + + Point decoding recognizes `x/y/z`, RGB or RGBA, and `nx/ny/nz`; all other + scalar properties become string-keyed point attributes. Gaussian decoding + interprets `f_dc_*`, opacity, scale, rotation, and ordered `f_rest_*` + properties. Header comments and the detected PLY format are retained in + cloud metadata. + + ## Arguments + + * `data` (`binary()`) — the complete PLY header and vertex body. + * `opts` (`keyword()`) — `:as` may be `:auto` (default), `:point_cloud`, or + `:gaussian_cloud`. `:auto` chooses Gaussian output when the property set + contains `f_dc_0`, `scale_0`, or `rot_0`; otherwise it chooses points. + The wire format is always read from the header. + + ## Returns + + * `{:ok, %PointCloud{}}` or `{:ok, %GaussianCloud{}}`, including metadata. + * `{:error, %ExCodecs.Error{reason: :invalid_data, codec: :ply}}` for + non-binary input, missing/invalid magic or format/header lines, missing + vertex elements, invalid counts/properties, unsupported list/property + types, too few ASCII rows, or a short binary body. + + ## Raises / exceptions + + The listed structural failures are returned. `Keyword.get/3` raises + `FunctionClauseError` when `opts` is not a proper keyword list. An unsupported + `:as` value raises `CaseClauseError`. Malformed row contents may also raise + (`KeyError`, `ArgumentError`, `FunctionClauseError`, or a bitstring match + exception), because scalar/required-property validation is not fully wrapped; + invalid ASCII scalar tokens are currently coerced to `0.0`. + + ## Examples + + iex> alias ExCodecs.Spatial.{Point, PointCloud} + iex> cloud = PointCloud.new([Point.new(1.0, 0.0, 0.0, normal: {0.0, 0.0, 1.0})]) + iex> {:ok, ply} = ExCodecs.Spatial.Codec.PLY.encode(cloud, ply_format: :binary_le) + iex> {:ok, %PointCloud{points: [point]}} = ExCodecs.Spatial.Codec.PLY.decode(ply) + iex> point.normal + {0.0, 0.0, 1.0} + """ + @spec decode(binary(), keyword()) :: + {:ok, PointCloud.t() | GaussianCloud.t()} | {:error, Error.t()} + def decode(data, opts \\ []) + + def decode(data, opts) when is_binary(data) do + as = Keyword.get(opts, :as, :auto) + + with {:ok, header, body} <- split_header(data), + {:ok, parsed} <- parse_header(header) do + decode_parsed_body(resolve_as(as, parsed.properties), body, parsed) + end + end + + def decode(_data, _opts) do + {:error, Error.new(:invalid_data, codec: :ply, message: "PLY data must be a binary")} + end + + @doc """ + Returns an enumerable over vertices from a PLY path or binary. + + Both modes currently read and decode the entire payload before yielding + elements; this is not incremental PLY parsing. + + ## Arguments + + * `source` (`Path.t() | binary()`) — a path or complete PLY payload. + * `opts` (`keyword()`) — `:source` may be `:auto` (default), `:file`, or + `:binary`. Auto mode treats a short path-like binary as a file only when + it names a regular file; otherwise it is payload data. `:as` has the same + meaning as in `decode/2`. + + ## Returns + + An `Enumerable.t()` yielding `%Point{}` or `%Gaussian{}`. Decode failures, + including `reason: :invalid_data`, are represented by exactly one + `{:error, %ExCodecs.Error{}}` element. A selected path that cannot be read + yields one error with `reason: :io_error` and the file reason in `details`. + + ## Raises / exceptions + + File and normal decode failures become stream elements. A non-binary + `source` raises `FunctionClauseError`. Keyword access raises + `FunctionClauseError` for invalid `opts`; unsupported `:source` values raise + `CaseClauseError`. Exceptions described by `decode/2` may occur while the + stream is initialized or enumerated. + + ## Examples + + iex> alias ExCodecs.Spatial.{Point, PointCloud} + iex> {:ok, ply} = ExCodecs.Spatial.Codec.PLY.encode(PointCloud.new([Point.new(0, 0, 0)])) + iex> [%Point{x: 0.0}] = ExCodecs.Spatial.Codec.PLY.stream_decode(ply) |> Enum.to_list() + """ + @spec stream_decode(binary() | Path.t(), keyword()) :: Enumerable.t() + def stream_decode(source, opts \\ []) + + def stream_decode(data, opts) when is_binary(data) do + case resolve_ply_source(data, opts) do + {:ok, :binary, bin} -> + stream_from_decoded_binary(bin, opts) + + {:ok, :file, path} -> + Stream.resource( + fn -> open_ply_file(path, opts) end, + &next_ply_item/1, + &close_ply_file/1 + ) + end + end + + defp stream_from_decoded_binary(bin, opts) do + case decode_to_list(bin, opts) do + {:ok, items} -> + Stream.map(items, & &1) + + {:error, error} -> + Stream.resource(fn -> {:error, error} end, &error_stream/1, fn _ -> :ok end) + end + end + + defp resolve_ply_source(bin, opts) do + case Keyword.get(opts, :source, :auto) do + :binary -> + {:ok, :binary, bin} + + :file -> + {:ok, :file, bin} + + :auto -> + if path_like_ply?(bin) and File.regular?(bin) do + {:ok, :file, bin} + else + {:ok, :binary, bin} + end + end + end + + defp path_like_ply?(bin) do + byte_size(bin) < 4096 and not ply_binary?(bin) and + (String.contains?(bin, "/") or String.contains?(bin, "\\") or + String.ends_with?(bin, [".ply", ".PLY"])) + end + + @doc false + def decode_header_and_body(data) when is_binary(data) do + with {:ok, header, body} <- split_header(data), + {:ok, parsed} <- parse_header(header) do + {:ok, parsed, body} + end + end + + # --- Header --------------------------------------------------------------- + + defp ply_binary?(<>) do + String.starts_with?(data, "ply") + end + + defp split_header(data) do + case :binary.match(data, "end_header") do + {pos, len} -> + # skip end_header and following newline(s) + rest = binary_part(data, pos + len, byte_size(data) - pos - len) + body = strip_leading_newlines(rest) + header = binary_part(data, 0, pos + len) + {:ok, header, body} + + :nomatch -> + {:error, Error.new(:invalid_data, codec: :ply, message: "PLY header missing end_header")} + end + end + + defp strip_leading_newlines(<<"\r\n", rest::binary>>), do: rest + defp strip_leading_newlines(<<"\n", rest::binary>>), do: rest + defp strip_leading_newlines(<<"\r", rest::binary>>), do: rest + defp strip_leading_newlines(bin), do: bin + + defp parse_header(header) do + lines = + header + |> String.split(~r/\r\n|\n|\r/, trim: true) + |> Enum.map(&String.trim/1) + + with :ok <- validate_magic(lines), + {:ok, format} <- find_format(lines), + {:ok, count, properties} <- find_vertex_element(lines) do + comments = + lines + |> Enum.filter(&String.starts_with?(&1, "comment ")) + |> Enum.map(&String.trim_leading(&1, "comment ")) + + {:ok, + %{ + format: format, + count: count, + properties: properties, + comments: comments + }} + end + end + + defp validate_magic(["ply" | _]), do: :ok + + defp validate_magic(_) do + {:error, Error.new(:invalid_data, codec: :ply, message: "PLY data must start with 'ply'")} + end + + defp find_format(lines) do + case Enum.find(lines, &String.starts_with?(&1, "format ")) do + "format ascii 1.0" <> _ -> + {:ok, :ascii} + + "format binary_little_endian 1.0" <> _ -> + {:ok, :binary_le} + + "format binary_big_endian 1.0" <> _ -> + {:ok, :binary_be} + + other when is_binary(other) -> + {:error, Error.new(:invalid_data, codec: :ply, message: "Unsupported PLY format: #{other}")} + + nil -> + {:error, Error.new(:invalid_data, codec: :ply, message: "PLY format line missing")} + end + end + + defp find_vertex_element(lines) do + case Enum.find_index(lines, &String.starts_with?(&1, "element vertex ")) do + nil -> + {:error, Error.new(:invalid_data, codec: :ply, message: "No vertex element in PLY")} + + idx -> + parse_vertex_element(lines, idx) + end + end + + defp parse_vertex_element(lines, idx) do + with {:ok, count} <- parse_vertex_count(Enum.at(lines, idx)), + {:ok, props} <- parse_vertex_properties(lines, idx) do + {:ok, count, props} + end + end + + defp parse_vertex_properties(lines, idx) do + props = + lines + |> Enum.drop(idx + 1) + |> Enum.take_while(&String.starts_with?(&1, "property ")) + |> Enum.map(&parse_property/1) + + if Enum.any?(props, &match?({:error, _}, &1)) do + {:error, Error.new(:invalid_data, codec: :ply, message: "Invalid PLY property")} + else + {:ok, props} + end + end + + defp parse_vertex_count(line) do + case String.split(line) do + ["element", "vertex", count_str] -> + case Integer.parse(count_str) do + {count, ""} when count >= 0 -> + {:ok, count} + + _ -> + {:error, + Error.new(:invalid_data, codec: :ply, message: "Invalid vertex count in PLY header")} + end + + _ -> + {:error, + Error.new(:invalid_data, codec: :ply, message: "Malformed element vertex line in PLY")} + end + end + + defp parse_property("property list " <> _), + do: {:error, :list_properties_unsupported} + + defp parse_property("property " <> rest) do + case String.split(rest) do + [type, name] -> + case property_type(type) do + {:ok, t} -> %{type: t, name: name} + :error -> {:error, :unknown_property_type} + end + + _ -> + {:error, :bad_property} + end + end + + defp property_type("char"), do: {:ok, :char} + defp property_type("uchar"), do: {:ok, :uchar} + defp property_type("short"), do: {:ok, :short} + defp property_type("ushort"), do: {:ok, :ushort} + defp property_type("int"), do: {:ok, :int} + defp property_type("uint"), do: {:ok, :uint} + defp property_type("float"), do: {:ok, :float} + defp property_type("double"), do: {:ok, :double} + defp property_type("int8"), do: {:ok, :char} + defp property_type("uint8"), do: {:ok, :uchar} + defp property_type("int16"), do: {:ok, :short} + defp property_type("uint16"), do: {:ok, :ushort} + defp property_type("int32"), do: {:ok, :int} + defp property_type("uint32"), do: {:ok, :uint} + defp property_type("float32"), do: {:ok, :float} + defp property_type("float64"), do: {:ok, :double} + defp property_type(_), do: :error + + # --- Encode points -------------------------------------------------------- + + defp point_properties(points) do + has_color? = Enum.any?(points, &Point.colored?/1) + has_alpha? = Enum.any?(points, fn p -> match?({_, _, _, _}, p.color) end) + has_normal? = Enum.any?(points, &Point.has_normal?/1) + + attr_names = + points + |> Enum.flat_map(fn p -> Map.keys(p.attributes) end) + |> Enum.uniq() + |> Enum.map(&to_string/1) + |> Enum.sort() + + base = [ + %{type: :float, name: "x"}, + %{type: :float, name: "y"}, + %{type: :float, name: "z"} + ] + + color = + cond do + has_alpha? -> + [ + %{type: :uchar, name: "red"}, + %{type: :uchar, name: "green"}, + %{type: :uchar, name: "blue"}, + %{type: :uchar, name: "alpha"} + ] + + has_color? -> + [ + %{type: :uchar, name: "red"}, + %{type: :uchar, name: "green"}, + %{type: :uchar, name: "blue"} + ] + + true -> + [] + end + + normal = + if has_normal? do + [ + %{type: :float, name: "nx"}, + %{type: :float, name: "ny"}, + %{type: :float, name: "nz"} + ] + else + [] + end + + attrs = Enum.map(attr_names, fn name -> %{type: :float, name: name} end) + {:ok, base ++ color ++ normal ++ attrs} + end + + defp gaussian_properties(gaussians) do + has_sh? = Enum.any?(gaussians, &(&1.sh != nil)) + + base = [ + %{type: :float, name: "x"}, + %{type: :float, name: "y"}, + %{type: :float, name: "z"}, + %{type: :float, name: "f_dc_0"}, + %{type: :float, name: "f_dc_1"}, + %{type: :float, name: "f_dc_2"}, + %{type: :float, name: "opacity"}, + %{type: :float, name: "scale_0"}, + %{type: :float, name: "scale_1"}, + %{type: :float, name: "scale_2"}, + %{type: :float, name: "rot_0"}, + %{type: :float, name: "rot_1"}, + %{type: :float, name: "rot_2"}, + %{type: :float, name: "rot_3"} + ] + + base ++ sh_property_defs(gaussians, has_sh?) + end + + defp sh_property_defs(_gaussians, false), do: [] + + defp sh_property_defs(gaussians, true) do + max_rest = + gaussians + |> Enum.map(&sh_rest_coeff_count/1) + |> Enum.max() + + for i <- 0..(max_rest - 1)//1, max_rest > 0 do + %{type: :float, name: "f_rest_#{i}"} + end + end + + defp sh_rest_coeff_count(%{sh: nil}), do: 0 + defp sh_rest_coeff_count(%{sh: [_dc | rest]}), do: length(List.flatten(rest)) + defp sh_rest_coeff_count(_), do: 0 + + defp build_header(format, count, props, comments) do + format_line = + case format do + :ascii -> "format ascii 1.0" + :binary_le -> "format binary_little_endian 1.0" + :binary_be -> "format binary_big_endian 1.0" + end + + comment_lines = Enum.map(comments, fn c -> "comment #{c}" end) + + prop_lines = + Enum.map(props, fn %{type: type, name: name} -> + "property #{type_name(type)} #{name}" + end) + + Enum.join( + ["ply", format_line] ++ + comment_lines ++ + ["element vertex #{count}"] ++ + prop_lines ++ + ["end_header", ""], + "\n" + ) + end + + # Encode only ever emits float and uchar properties (attributes are promoted + # to float32; see docs/spatial_formats.md), so no other types are needed here. + defp type_name(:uchar), do: "uchar" + defp type_name(:float), do: "float" + + defp encode_points(points, props, :ascii) do + points + |> Enum.map_join("\n", fn point -> + point + |> point_values(props) + |> Enum.map_join(" ", &ascii_value/1) + end) + |> then(&(&1 <> "\n")) + end + + defp encode_points(points, props, endian) when endian in [:binary_le, :binary_be] do + IO.iodata_to_binary( + Enum.map(points, fn point -> + point + |> point_values(props) + |> Enum.zip(props) + |> Enum.map(fn {value, %{type: type}} -> pack(type, value, endian) end) + end) + ) + end + + defp point_values(%Point{} = p, props) do + {nx, ny, nz} = p.normal || {0.0, 0.0, 0.0} + + known = %{ + "x" => p.x, + "y" => p.y, + "z" => p.z, + "nx" => nx, + "ny" => ny, + "nz" => nz, + "red" => color_channel(p.color, 0), + "green" => color_channel(p.color, 1), + "blue" => color_channel(p.color, 2), + "alpha" => color_channel(p.color, 3, 255) + } + + Enum.map(props, fn %{name: name} -> attribute_value(known, p.attributes, name) end) + end + + defp attribute_value(known, attributes, name) do + cond do + Map.has_key?(known, name) -> Map.fetch!(known, name) + Map.has_key?(attributes, name) -> Map.fetch!(attributes, name) + true -> 0.0 + end + end + + defp color_channel(color, idx, default \\ 0) + defp color_channel(nil, _idx, default), do: default + + defp color_channel(color, idx, default) do + if idx < tuple_size(color), do: elem(color, idx), else: default + end + + defp encode_gaussians(gaussians, props, :ascii) do + gaussians + |> Enum.map_join("\n", fn g -> + g + |> gaussian_values(props) + |> Enum.map_join(" ", &ascii_value/1) + end) + |> then(&(&1 <> "\n")) + end + + defp encode_gaussians(gaussians, props, endian) when endian in [:binary_le, :binary_be] do + IO.iodata_to_binary( + Enum.map(gaussians, fn g -> + g + |> gaussian_values(props) + |> Enum.zip(props) + |> Enum.map(fn {value, %{type: type}} -> pack(type, value, endian) end) + end) + ) + end + + defp gaussian_values(%Gaussian{} = g, props) do + {x, y, z} = g.position + {r, gc, b} = g.color + {sx, sy, sz} = g.scale + {rw, rx, ry, rz} = g.rotation + rest = sh_rest_flat(g.sh) + + known = %{ + "x" => x, + "y" => y, + "z" => z, + "f_dc_0" => r, + "f_dc_1" => gc, + "f_dc_2" => b, + "opacity" => g.opacity, + "scale_0" => sx, + "scale_1" => sy, + "scale_2" => sz, + "rot_0" => rw, + "rot_1" => rx, + "rot_2" => ry, + "rot_3" => rz + } + + Enum.map(props, fn %{name: name} -> + case Map.fetch(known, name) do + {:ok, value} -> value + :error -> rest_coeff(name, rest) + end + end) + end + + defp rest_coeff("f_rest_" <> idx, rest), do: Enum.at(rest, String.to_integer(idx), 0.0) + + defp sh_rest_flat(nil), do: [] + defp sh_rest_flat([_dc | rest]), do: List.flatten(rest) + defp sh_rest_flat(other) when is_list(other), do: List.flatten(other) + + defp ascii_value(v) when is_integer(v), do: Integer.to_string(v) + defp ascii_value(v) when is_float(v), do: :erlang.float_to_binary(v, [:short]) + + # Like type_name/1, encode only packs float and uchar values; decode's + # unpack/3 still handles the full range of PLY scalar property types. + defp pack(:uchar, v, _), do: <> + defp pack(:float, v, :binary_le), do: <> + defp pack(:float, v, :binary_be), do: <> + + # --- Decode --------------------------------------------------------------- + + defp resolve_as(:point_cloud, _), do: :point_cloud + defp resolve_as(:gaussian_cloud, _), do: :gaussian_cloud + + defp resolve_as(:auto, props) do + names = MapSet.new(Enum.map(props, & &1.name)) + + if MapSet.member?(names, "f_dc_0") or MapSet.member?(names, "scale_0") or + MapSet.member?(names, "rot_0") do + :gaussian_cloud + else + :point_cloud + end + end + + defp decode_parsed_body(:point_cloud, body, parsed) do + with {:ok, points} <- decode_vertices(body, parsed) do + {:ok, PointCloud.new(points, metadata: ply_metadata(parsed))} + end + end + + defp decode_parsed_body(:gaussian_cloud, body, parsed) do + with {:ok, gaussians} <- decode_gaussians(body, parsed) do + {:ok, GaussianCloud.new(gaussians, metadata: ply_metadata(parsed))} + end + end + + defp ply_metadata(parsed) do + Metadata.new( + comments: parsed.comments, + entries: %{"ply_format" => to_string(parsed.format)} + ) + end + + defp decode_vertices(body, %{format: :ascii, count: count, properties: props}) do + lines = + body + |> String.split(~r/\r\n|\n|\r/, trim: true) + |> Enum.take(count) + + if length(lines) < count do + {:error, + Error.new(:invalid_data, + codec: :ply, + message: "Expected #{count} vertices, got #{length(lines)}" + )} + else + points = + Enum.map(lines, fn line -> + values = String.split(line) |> Enum.map(&parse_ascii_number/1) + values_to_point(values, props) + end) + + {:ok, points} + end + end + + defp decode_vertices(body, %{format: endian, count: count, properties: props}) + when endian in [:binary_le, :binary_be] do + stride = Enum.reduce(props, 0, fn p, acc -> acc + type_size(p.type) end) + expected = stride * count + + if byte_size(body) < expected do + {:error, + Error.new(:invalid_data, + codec: :ply, + message: "Binary PLY body too short" + )} + else + {points, _} = + Enum.map_reduce(1..count, body, fn _, rest -> + {values, next} = unpack_row(rest, props, endian) + {values_to_point(values, props), next} + end) + + {:ok, points} + end + end + + defp decode_gaussians(body, parsed) do + with {:ok, points} <- decode_vertices(body, parsed) do + gaussians = + Enum.map(points, fn %Point{} = p -> + # Point.new/4 normalizes attribute keys to strings. + attrs = p.attributes + + Gaussian.new({p.x, p.y, p.z}, + color: { + Map.get(attrs, "f_dc_0", 0.5), + Map.get(attrs, "f_dc_1", 0.5), + Map.get(attrs, "f_dc_2", 0.5) + }, + opacity: Map.get(attrs, "opacity", 1.0), + scale: { + Map.get(attrs, "scale_0", 1.0), + Map.get(attrs, "scale_1", 1.0), + Map.get(attrs, "scale_2", 1.0) + }, + rotation: { + Map.get(attrs, "rot_0", 1.0), + Map.get(attrs, "rot_1", 0.0), + Map.get(attrs, "rot_2", 0.0), + Map.get(attrs, "rot_3", 0.0) + }, + sh: extract_sh(attrs), + metadata: attrs + ) + end) + + {:ok, gaussians} + end + end + + defp extract_sh(attrs) do + rest_keys = + attrs + |> Map.keys() + |> Enum.filter(&String.starts_with?(&1, "f_rest_")) + |> Enum.sort_by(fn "f_rest_" <> i -> String.to_integer(i) end) + + if rest_keys == [] do + nil + else + dc = [ + Map.get(attrs, "f_dc_0", 0.0), + Map.get(attrs, "f_dc_1", 0.0), + Map.get(attrs, "f_dc_2", 0.0) + ] + + rest = Enum.map(rest_keys, &Map.get(attrs, &1, 0.0)) + [dc | Enum.chunk_every(rest, 3)] + end + end + + defp values_to_point(values, props) do + paired = Enum.zip(props, values) |> Map.new(fn {%{name: n}, v} -> {n, v} end) + + color = + cond do + Map.has_key?(paired, "alpha") and Map.has_key?(paired, "red") -> + {trunc(paired["red"]), trunc(paired["green"]), trunc(paired["blue"]), + trunc(paired["alpha"])} + + Map.has_key?(paired, "red") -> + {trunc(paired["red"]), trunc(paired["green"]), trunc(paired["blue"])} + + true -> + nil + end + + normal = + if Map.has_key?(paired, "nx") do + {paired["nx"] * 1.0, paired["ny"] * 1.0, paired["nz"] * 1.0} + else + nil + end + + known = ["x", "y", "z", "red", "green", "blue", "alpha", "nx", "ny", "nz"] + + attributes = + paired + |> Enum.reject(fn {key, _value} -> key in known end) + |> Map.new() + + Point.new(paired["x"] || 0.0, paired["y"] || 0.0, paired["z"] || 0.0, + color: color, + normal: normal, + attributes: attributes + ) + end + + defp parse_ascii_number(str) do + case Integer.parse(str) do + {i, ""} -> + i + + _ -> + case Float.parse(str) do + {f, _} -> f + :error -> 0.0 + end + end + end + + defp unpack_row(bin, props, endian) do + Enum.map_reduce(props, bin, fn %{type: type}, rest -> + unpack(type, rest, endian) + end) + end + + defp unpack(:uchar, <>, _), do: {v, rest} + defp unpack(:char, <>, _), do: {v, rest} + + defp unpack(:ushort, <>, :binary_le), + do: {v, rest} + + defp unpack(:ushort, <>, :binary_be), do: {v, rest} + defp unpack(:short, <>, :binary_le), do: {v, rest} + defp unpack(:short, <>, :binary_be), do: {v, rest} + defp unpack(:uint, <>, :binary_le), do: {v, rest} + defp unpack(:uint, <>, :binary_be), do: {v, rest} + defp unpack(:int, <>, :binary_le), do: {v, rest} + defp unpack(:int, <>, :binary_be), do: {v, rest} + defp unpack(:float, <>, :binary_le), do: {v, rest} + defp unpack(:float, <>, :binary_be), do: {v, rest} + defp unpack(:double, <>, :binary_le), do: {v, rest} + defp unpack(:double, <>, :binary_be), do: {v, rest} + + defp type_size(:char), do: 1 + defp type_size(:uchar), do: 1 + defp type_size(:short), do: 2 + defp type_size(:ushort), do: 2 + defp type_size(:int), do: 4 + defp type_size(:uint), do: 4 + defp type_size(:float), do: 4 + defp type_size(:double), do: 8 + + defp ply_format_opt(opts) do + Keyword.get(opts, :ply_format) || Keyword.get(opts, :format, :ascii) + end + + defp normalize_format(:ascii), do: :ascii + defp normalize_format(:binary), do: :binary_le + defp normalize_format(:binary_le), do: :binary_le + defp normalize_format(:binary_be), do: :binary_be + defp normalize_format(:little), do: :binary_le + defp normalize_format(:big), do: :binary_be + defp normalize_format(_), do: :ascii + + # --- Streaming helpers ---------------------------------------------------- + + defp decode_to_list(data, opts) when is_binary(data) do + case decode(data, opts) do + {:ok, %PointCloud{points: points}} -> {:ok, points} + {:ok, %GaussianCloud{gaussians: gs}} -> {:ok, gs} + {:error, _} = err -> err + end + end + + defp open_ply_file(path, opts) do + case File.read(path) do + {:ok, data} -> + case decode_to_list(data, opts) do + {:ok, items} -> {:ok, items} + {:error, error} -> {:error, error} + end + + {:error, reason} -> + {:error, + Error.new(:io_error, + codec: :ply, + message: "Failed to read PLY file: #{inspect(reason)}", + details: reason + )} + end + end + + defp next_ply_item({:ok, [item | rest]}), do: {[item], {:ok, rest}} + defp next_ply_item({:ok, []}), do: {:halt, :done} + defp next_ply_item({:error, error}), do: {[{:error, error}], :done} + defp next_ply_item(:done), do: {:halt, :done} + + defp close_ply_file(_), do: :ok + + defp error_stream({:error, error}), do: {[{:error, error}], :done} + defp error_stream(:done), do: {:halt, :done} +end diff --git a/lib/ex_codecs/spatial/gaussian.ex b/lib/ex_codecs/spatial/gaussian.ex new file mode 100644 index 0000000..6e4e183 --- /dev/null +++ b/lib/ex_codecs/spatial/gaussian.ex @@ -0,0 +1,140 @@ +defmodule ExCodecs.Spatial.Gaussian do + @moduledoc """ + A single 3D Gaussian splat used for data interchange, not rendering. + + ## Fields + + * `:position` — `{float(), float(), float()}` center in application-defined + Cartesian units; required by the struct and defaults to the origin. + * `:rotation` — `quaternion()` in scalar-first `{w, x, y, z}` order; + defaults to identity `{1.0, 0.0, 0.0, 0.0}`. Normalization is not enforced. + * `:scale` — `scale3()` along local axes in the position's units; defaults + to `{1.0, 1.0, 1.0}`. Positivity is not enforced. + * `:opacity` — `float()`; defaults to `1.0`, conventionally in `0.0..1.0`. + * `:color` — `rgb()` linear RGB; defaults to mid-gray + `{0.5, 0.5, 0.5}`. Components are commonly DC color coefficients. + * `:sh` — `sh_coeffs()`; defaults to `nil`. Coefficient ordering is + codec-specific. + * `:metadata` — `map()` of application data; defaults to `%{}`. + + ## Example + + iex> ExCodecs.Spatial.Gaussian.new({1, 2, 3}, + ...> rotation: {1, 0, 0, 0}, + ...> scale: {0.1, 0.2, 0.3}, + ...> opacity: 0.8, + ...> color: {1.0, 0.25, 0.0} + ...> ) + %ExCodecs.Spatial.Gaussian{ + position: {1.0, 2.0, 3.0}, + rotation: {1.0, 0.0, 0.0, 0.0}, + scale: {0.1, 0.2, 0.3}, + opacity: 0.8, + color: {1.0, 0.25, 0.0}, + sh: nil, + metadata: %{} + } + """ + + @typedoc """ + Linear RGB components `{red, green, blue}`. Values are conventionally + normalized but no range is enforced. For example, `{1.0, 0.25, 0.0}` is a + warm orange. + """ + @type rgb :: {float(), float(), float()} + @typedoc """ + A scalar-first rotation quaternion `{w, x, y, z}`. Unit length is + conventional but is not enforced. For example, + `{1.0, 0.0, 0.0, 0.0}` is the identity rotation. + """ + @type quaternion :: {float(), float(), float(), float()} + @typedoc """ + Local-axis Gaussian scales `{sx, sy, sz}` in position units. + For example, `{0.1, 0.2, 0.05}` describes an anisotropic Gaussian. + """ + @type scale3 :: {float(), float(), float()} + @typedoc """ + Codec-specific nested spherical-harmonic coefficient lists, or `nil` when + no coefficients are present. For example, `[[0.1, 0.2, 0.3]]` stores one + RGB coefficient group. + """ + @type sh_coeffs :: [[float()]] | nil + + @typedoc """ + A Gaussian splat. See the module documentation for every field, its default, + units or conventions, and a construction example. + """ + @type t :: %__MODULE__{ + position: {float(), float(), float()}, + rotation: quaternion(), + scale: scale3(), + opacity: float(), + color: rgb(), + sh: sh_coeffs(), + metadata: map() + } + + @enforce_keys [:position] + defstruct position: {0.0, 0.0, 0.0}, + rotation: {1.0, 0.0, 0.0, 0.0}, + scale: {1.0, 1.0, 1.0}, + opacity: 1.0, + color: {0.5, 0.5, 0.5}, + sh: nil, + metadata: %{} + + @doc """ + Builds a Gaussian from position and options. + + ## Arguments + + * `position` — `{number(), number(), number()}` center coordinates, stored + as floats. + * `opts` — a keyword list with: + * `:rotation` — numeric `{w, x, y, z}`; defaults to identity. + * `:scale` — numeric `{sx, sy, sz}`; defaults to `{1.0, 1.0, 1.0}`. + * `:opacity` — number; defaults to `1.0` and is stored as a float. + * `:color` — numeric `{r, g, b}`; defaults to `{0.5, 0.5, 0.5}`. + * `:sh` — `sh_coeffs()`; defaults to `nil`. + * `:metadata` — `map()`; defaults to `%{}`. + + ## Returns + + `%Gaussian{}` + + ## Raises + + * `FunctionClauseError` for a non-numeric or malformed position, rotation, + or scale/color tuple, or when `opts` is not a keyword list. + * `ArithmeticError` if `:opacity` is not numeric. + + ## Examples + + iex> g = ExCodecs.Spatial.Gaussian.new({1.0, 2.0, 3.0}, opacity: 0.5) + iex> g.position + {1.0, 2.0, 3.0} + iex> g.opacity + 0.5 + """ + @spec new({number(), number(), number()}, keyword()) :: t() + def new({x, y, z}, opts \\ []) when is_number(x) and is_number(y) and is_number(z) do + %__MODULE__{ + position: {x * 1.0, y * 1.0, z * 1.0}, + rotation: float4(Keyword.get(opts, :rotation, {1.0, 0.0, 0.0, 0.0})), + scale: float3(Keyword.get(opts, :scale, {1.0, 1.0, 1.0})), + opacity: Keyword.get(opts, :opacity, 1.0) * 1.0, + color: float3(Keyword.get(opts, :color, {0.5, 0.5, 0.5})), + sh: Keyword.get(opts, :sh), + metadata: Keyword.get(opts, :metadata, %{}) + } + end + + defp float3({a, b, c}) when is_number(a) and is_number(b) and is_number(c) do + {a * 1.0, b * 1.0, c * 1.0} + end + + defp float4({a, b, c, d}) + when is_number(a) and is_number(b) and is_number(c) and is_number(d) do + {a * 1.0, b * 1.0, c * 1.0, d * 1.0} + end +end diff --git a/lib/ex_codecs/spatial/gaussian_cloud.ex b/lib/ex_codecs/spatial/gaussian_cloud.ex new file mode 100644 index 0000000..a9eb496 --- /dev/null +++ b/lib/ex_codecs/spatial/gaussian_cloud.ex @@ -0,0 +1,135 @@ +defmodule ExCodecs.Spatial.GaussianCloud do + @moduledoc """ + A collection of `%ExCodecs.Spatial.Gaussian{}` values for codec interchange. + + ## Fields + + * `:gaussians` — `[Gaussian.t()]`; defaults to `[]`. Ordering is preserved. + Positions use application-defined Cartesian units. + * `:bounds` — `Bounds.t() | nil`; defaults to `nil`. `new/2` computes + axis-aligned bounds from Gaussian centers by default; extents do not + include each Gaussian's scale. + * `:metadata` — `Metadata.t()`; defaults to `%Metadata{}`. + + ## Example + + iex> alias ExCodecs.Spatial.{Gaussian, GaussianCloud} + iex> cloud = GaussianCloud.new([Gaussian.new({-1, 0, 0}), Gaussian.new({2, 0, 0})]) + iex> {GaussianCloud.size(cloud), cloud.bounds.min_x, cloud.bounds.max_x} + {2, -1.0, 2.0} + """ + + alias ExCodecs.Spatial.{Bounds, Gaussian, Metadata} + + @typedoc """ + A Gaussian cloud. See the module documentation for every field, its default, + units or conventions, and a construction example. + """ + @type t :: %__MODULE__{ + gaussians: [Gaussian.t()], + bounds: Bounds.t() | nil, + metadata: Metadata.t() + } + + defstruct gaussians: [], + bounds: nil, + metadata: %Metadata{} + + @doc """ + Builds a Gaussian cloud. + + ## Arguments + + * `gaussians` — `[Gaussian.t()]`; the supplied list is retained. + * `opts` — a keyword list with: + * `:compute_bounds` — any truthy/falsy term; defaults to `true`. + * `:bounds` — `Bounds.t() | nil`; a truthy value overrides computation. + * `:metadata` — `Metadata.t()`; defaults to `%Metadata{}`. + + ## Returns + + `%GaussianCloud{}` + + ## Raises + + * `FunctionClauseError` if `gaussians` is not a list, an extracted position + is not a numeric 3-tuple, or `opts` is not a keyword list. + * `KeyError` or `BadMapError` if an item does not expose `:position`. + + ## Examples + + iex> alias ExCodecs.Spatial.{Gaussian, GaussianCloud} + iex> c = GaussianCloud.new([Gaussian.new({0.0, 0.0, 0.0})]) + iex> GaussianCloud.size(c) + 1 + """ + @spec new([Gaussian.t()], keyword()) :: t() + def new(gaussians, opts \\ []) when is_list(gaussians) do + compute_bounds? = Keyword.get(opts, :compute_bounds, true) + metadata = Keyword.get(opts, :metadata, %Metadata{}) + bounds = Keyword.get(opts, :bounds) + + positions = Enum.map(gaussians, & &1.position) + + %__MODULE__{ + gaussians: gaussians, + bounds: bounds || if(compute_bounds?, do: Bounds.from_points(positions), else: nil), + metadata: metadata + } + end + + @doc """ + Number of Gaussians. + + ## Arguments + + * `cloud` — `%GaussianCloud{}` + + ## Returns + + non-negative integer + + ## Raises + + * `FunctionClauseError` if `cloud` is not a `%GaussianCloud{}`. + * `ArgumentError` if a manually constructed cloud has a non-list + `:gaussians` field. + + ## Examples + + iex> ExCodecs.Spatial.GaussianCloud.size(ExCodecs.Spatial.GaussianCloud.new([])) + 0 + """ + @spec size(t()) :: non_neg_integer() + def size(%__MODULE__{gaussians: gaussians}), do: length(gaussians) + + @doc """ + Recomputes bounds from Gaussian positions. + + ## Arguments + + * `cloud` — `%GaussianCloud{}` + + ## Returns + + Updated `%GaussianCloud{}` + + ## Raises + + * `FunctionClauseError` if `cloud` is not a `%GaussianCloud{}` or an + extracted position is not a numeric 3-tuple. + * `KeyError` or `BadMapError` if an item does not expose `:position`. + + ## Examples + + iex> alias ExCodecs.Spatial.{Gaussian, GaussianCloud} + iex> c = GaussianCloud.new([Gaussian.new({1.0, 2.0, 3.0})], compute_bounds: false) + iex> GaussianCloud.with_bounds(c).bounds != nil + true + """ + @spec with_bounds(t()) :: t() + def with_bounds(%__MODULE__{gaussians: gaussians} = cloud) do + positions = Enum.map(gaussians, & &1.position) + %{cloud | bounds: Bounds.from_points(positions)} + end +end diff --git a/lib/ex_codecs/spatial/metadata.ex b/lib/ex_codecs/spatial/metadata.ex new file mode 100644 index 0000000..ebfe7c7 --- /dev/null +++ b/lib/ex_codecs/spatial/metadata.ex @@ -0,0 +1,176 @@ +defmodule ExCodecs.Spatial.Metadata do + @moduledoc """ + Free-form metadata for spatial clouds. + + ## Fields + + * `:entries` — `%{optional(String.t()) => term()}` arbitrary format or + application values; defaults to `%{}`. String keys are the public API + convention. + * `:comments` — `[String.t()]` ordered human-readable comments; defaults + to `[]`. No encoding beyond Elixir UTF-8 string conventions is imposed. + * `:source` — `String.t() | nil` source file, URI, sensor, or producer + identifier; defaults to `nil`. + * `:created_at` — `DateTime.t() | nil` creation instant; defaults to `nil`. + Use a timezone-aware `DateTime`, conventionally UTC. + + ## Example + + iex> created_at = ~U[2026-07-16 12:00:00Z] + iex> ExCodecs.Spatial.Metadata.new( + ...> entries: %{"coordinate_system" => "ENU"}, + ...> comments: ["Captured after calibration"], + ...> source: "sensor://roof-lidar", + ...> created_at: created_at + ...> ) + %ExCodecs.Spatial.Metadata{ + entries: %{"coordinate_system" => "ENU"}, + comments: ["Captured after calibration"], + source: "sensor://roof-lidar", + created_at: ~U[2026-07-16 12:00:00Z] + } + """ + + @typedoc """ + Spatial metadata. See the module documentation for every field, its default, + units or conventions, and a construction example. + """ + @type t :: %__MODULE__{ + entries: %{optional(String.t()) => term()}, + comments: [String.t()], + source: String.t() | nil, + created_at: DateTime.t() | nil + } + + defstruct entries: %{}, + comments: [], + source: nil, + created_at: nil + + @doc """ + Builds metadata from options. + + ## Arguments + + * `opts` — a keyword list with: + * `:entries` — an enumerable accepted by `Map.new/1`, conventionally a + map with string keys; defaults to `%{}`. + * `:comments` — `[String.t()]`; defaults to `[]`. + * `:source` — `String.t() | nil`; defaults to `nil`. + * `:created_at` — `DateTime.t() | nil`; defaults to `nil`. + + ## Returns + + `%Metadata{}` + + ## Raises + + * `FunctionClauseError` if `opts` is not a keyword list. + * `Protocol.UndefinedError` if `:entries` is not enumerable. + * `ArgumentError` if an enumerable entry cannot be converted to a map pair. + + ## Examples + + iex> m = ExCodecs.Spatial.Metadata.new(comments: ["hi"], entries: %{"k" => 1}) + iex> m.comments + ["hi"] + """ + @spec new(keyword()) :: t() + def new(opts \\ []) do + %__MODULE__{ + entries: Map.new(Keyword.get(opts, :entries, %{})), + comments: Keyword.get(opts, :comments, []), + source: Keyword.get(opts, :source), + created_at: Keyword.get(opts, :created_at) + } + end + + @doc """ + Puts a string key into `entries`. + + ## Arguments + + * `meta` — `%Metadata{}` + * `key` — `String.t()` key. + * `value` — `term()` to store, replacing an existing value. + + ## Returns + + Updated `%Metadata{}` + + ## Raises + + * `FunctionClauseError` unless `meta` is `%Metadata{}` and `key` is a binary. + * `BadMapError` if a manually constructed metadata value has a non-map + `:entries` field. + + ## Examples + + iex> m = ExCodecs.Spatial.Metadata.put(ExCodecs.Spatial.Metadata.new(), "a", 1) + iex> ExCodecs.Spatial.Metadata.get(m, "a") + 1 + """ + @spec put(t(), String.t(), term()) :: t() + def put(%__MODULE__{} = meta, key, value) when is_binary(key) do + %{meta | entries: Map.put(meta.entries, key, value)} + end + + @doc """ + Appends a comment string. + + ## Arguments + + * `meta` — `%Metadata{}` + * `comment` — `String.t()` appended after existing comments. + + ## Returns + + Updated `%Metadata{}` + + ## Raises + + * `FunctionClauseError` unless `meta` is `%Metadata{}` and `comment` is a + binary. + * `ArgumentError` if a manually constructed metadata value has an improper + `:comments` list. + + ## Examples + + iex> m = ExCodecs.Spatial.Metadata.add_comment(ExCodecs.Spatial.Metadata.new(), "x") + iex> m.comments + ["x"] + """ + @spec add_comment(t(), String.t()) :: t() + def add_comment(%__MODULE__{} = meta, comment) when is_binary(comment) do + %{meta | comments: meta.comments ++ [comment]} + end + + @doc """ + Fetches an entry by string key. + + ## Arguments + + * `meta` — `%Metadata{}` + * `key` — `String.t()` to look up. + * `default` — `term()` returned when the key is absent; defaults to `nil`. + + ## Returns + + Stored value or `default` + + ## Raises + + * `FunctionClauseError` unless `meta` is `%Metadata{}` and `key` is a binary. + * `BadMapError` if a manually constructed metadata value has a non-map + `:entries` field. + + ## Examples + + iex> ExCodecs.Spatial.Metadata.get(ExCodecs.Spatial.Metadata.new(), "missing", :nope) + :nope + """ + @spec get(t(), String.t(), term()) :: term() + def get(%__MODULE__{entries: entries}, key, default \\ nil) when is_binary(key) do + Map.get(entries, key, default) + end +end diff --git a/lib/ex_codecs/spatial/point.ex b/lib/ex_codecs/spatial/point.ex new file mode 100644 index 0000000..1cba240 --- /dev/null +++ b/lib/ex_codecs/spatial/point.ex @@ -0,0 +1,205 @@ +defmodule ExCodecs.Spatial.Point do + @moduledoc """ + A 3D point with optional color, surface normal, and attributes. + + ## Fields + + * `:x`, `:y`, `:z` — `float()` Cartesian coordinates. `new/4` requires + them and converts numbers to floats; the struct defaults are `0.0`. + Units are application-defined and must be consistent within a cloud. + * `:color` — `rgb() | rgba() | nil`; defaults to `nil`. Channels are + non-negative integers conventionally in `0..255`; alpha is last. + * `:normal` — `normal() | nil`; defaults to `nil`. Components follow the + same axis convention as the coordinates and are normally unit length, + though this module does not normalize them. + * `:attributes` — `attributes()`; defaults to `%{}`. Keys are **strings** + (atoms passed to `new/4` are converted via `to_string/1`). Values are + numbers or binaries. + + ## Example + + iex> ExCodecs.Spatial.Point.new(1, 2.5, -3, color: {255, 128, 0}, normal: {0.0, 0.0, 1.0}) + %ExCodecs.Spatial.Point{ + x: 1.0, + y: 2.5, + z: -3.0, + color: {255, 128, 0}, + normal: {0.0, 0.0, 1.0}, + attributes: %{} + } + """ + + @typedoc """ + An RGB color tuple `{red, green, blue}`. Channels are non-negative integers, + conventionally in `0..255`; that range is not enforced. For example, + `{255, 128, 0}` represents orange. + """ + @type rgb :: {non_neg_integer(), non_neg_integer(), non_neg_integer()} + @typedoc """ + An RGBA color tuple `{red, green, blue, alpha}`. Channels are non-negative + integers, conventionally in `0..255`; that range is not enforced. For + example, `{0, 64, 255, 128}` is a half-transparent blue. + """ + @type rgba :: {non_neg_integer(), non_neg_integer(), non_neg_integer(), non_neg_integer()} + @typedoc """ + A surface-normal vector `{nx, ny, nz}` in the point coordinate convention. + Unit length is conventional but is not enforced. For example, + `{0.0, 0.0, 1.0}` points along the positive Z axis. + """ + @type normal :: {float(), float(), float()} + @typedoc """ + User attributes keyed by strings, with numeric or binary values. + For example, `%{"intensity" => 0.82, "classification" => 2}`. + Atom keys passed to `new/4` are normalized to strings. + """ + @type attributes :: %{optional(String.t()) => number() | binary()} + + @typedoc """ + A spatial point. See the module documentation for every field, its default, + units or conventions, and a construction example. + """ + @type t :: %__MODULE__{ + x: float(), + y: float(), + z: float(), + color: rgb() | rgba() | nil, + normal: normal() | nil, + attributes: attributes() + } + + @enforce_keys [:x, :y, :z] + defstruct x: 0.0, + y: 0.0, + z: 0.0, + color: nil, + normal: nil, + attributes: %{} + + @doc """ + Builds a point from coordinates and optional fields. + + ## Arguments + + * `x`, `y`, `z` — numbers (stored as floats) + * `opts`: + * `:color` — `rgb` or `rgba` tuple, or `nil` + * `:normal` — `{nx, ny, nz}` or `nil` + * `:attributes` — map of string keys (default `%{}`); atom keys are + stringified + + ## Returns + + `%ExCodecs.Spatial.Point{}` + + ## Raises + + * `FunctionClauseError` if a coordinate is not numeric, or if `opts` is not + a keyword list accepted by `Keyword.get/3`. + + ## Examples + + iex> p = ExCodecs.Spatial.Point.new(1.0, 2.0, 3.0) + iex> {p.x, p.y, p.z} + {1.0, 2.0, 3.0} + + iex> p = ExCodecs.Spatial.Point.new(0.0, 0.0, 0.0, color: {255, 128, 0}) + iex> p.color + {255, 128, 0} + """ + @spec new(number(), number(), number(), keyword()) :: t() + def new(x, y, z, opts \\ []) when is_number(x) and is_number(y) and is_number(z) do + %__MODULE__{ + x: x * 1.0, + y: y * 1.0, + z: z * 1.0, + color: Keyword.get(opts, :color), + normal: Keyword.get(opts, :normal), + attributes: normalize_attributes(Keyword.get(opts, :attributes, %{})) + } + end + + defp normalize_attributes(attrs) when is_map(attrs) do + Map.new(attrs, fn + {k, v} when is_binary(k) -> {k, v} + {k, v} -> {to_string(k), v} + end) + end + + @doc """ + Returns `{x, y, z}`. + + ## Arguments + + * `point` — `%Point{}` + + ## Returns + + `{float(), float(), float()}` + + ## Raises + + * `FunctionClauseError` if `point` is not a `%Point{}`. + + ## Examples + + iex> ExCodecs.Spatial.Point.coords(ExCodecs.Spatial.Point.new(1, 2, 3)) + {1.0, 2.0, 3.0} + """ + @spec coords(t()) :: {float(), float(), float()} + def coords(%__MODULE__{x: x, y: y, z: z}), do: {x, y, z} + + @doc """ + Returns whether the point has RGB or RGBA color. + + ## Arguments + + * `point` — `%Point{}` + + ## Returns + + `true` | `false` + + ## Raises + + * `FunctionClauseError` if the argument is not a `%Point{}` or its `:color` + is neither `nil`, a 3-tuple, nor a 4-tuple. + + ## Examples + + iex> ExCodecs.Spatial.Point.colored?(ExCodecs.Spatial.Point.new(0, 0, 0)) + false + iex> ExCodecs.Spatial.Point.colored?(ExCodecs.Spatial.Point.new(0, 0, 0, color: {1, 2, 3})) + true + """ + @spec colored?(t()) :: boolean() + def colored?(%__MODULE__{color: nil}), do: false + def colored?(%__MODULE__{color: {_r, _g, _b}}), do: true + def colored?(%__MODULE__{color: {_r, _g, _b, _a}}), do: true + + @doc """ + Returns whether the point has a surface normal. + + ## Arguments + + * `point` — `%Point{}` + + ## Returns + + `true` | `false` + + ## Raises + + * `FunctionClauseError` if the argument is not a `%Point{}` or its `:normal` + is neither `nil` nor a 3-tuple. + + ## Examples + + iex> ExCodecs.Spatial.Point.has_normal?(ExCodecs.Spatial.Point.new(0, 0, 0)) + false + iex> ExCodecs.Spatial.Point.has_normal?(ExCodecs.Spatial.Point.new(0, 0, 0, normal: {0.0, 1.0, 0.0})) + true + """ + @spec has_normal?(t()) :: boolean() + def has_normal?(%__MODULE__{normal: nil}), do: false + def has_normal?(%__MODULE__{normal: {_nx, _ny, _nz}}), do: true +end diff --git a/lib/ex_codecs/spatial/point_cloud.ex b/lib/ex_codecs/spatial/point_cloud.ex new file mode 100644 index 0000000..506ce26 --- /dev/null +++ b/lib/ex_codecs/spatial/point_cloud.ex @@ -0,0 +1,259 @@ +defmodule ExCodecs.Spatial.PointCloud do + @moduledoc """ + A collection of `%ExCodecs.Spatial.Point{}` values with bounds and metadata. + + ## Fields + + * `:points` — `[Point.t()]`; defaults to `[]`. Ordering is preserved. + Coordinate units and conventions are defined by the application. + * `:bounds` — `Bounds.t() | nil`; defaults to `nil`. `new/2` computes an + axis-aligned box by default; `nil` also represents an empty cloud or + deliberately uncomputed bounds. + * `:metadata` — `Metadata.t()`; defaults to `%Metadata{}`. + + ## Example + + iex> alias ExCodecs.Spatial.{Point, PointCloud} + iex> cloud = PointCloud.new([Point.new(0, 0, 0), Point.new(2, 1, -1)]) + iex> {PointCloud.size(cloud), cloud.bounds.max_x} + {2, 2.0} + """ + + alias ExCodecs.Spatial.{Bounds, Metadata, Point} + + @typedoc """ + A point cloud. See the module documentation for every field, its default, + units or conventions, and a construction example. + """ + @type t :: %__MODULE__{ + points: [Point.t()], + bounds: Bounds.t() | nil, + metadata: Metadata.t() + } + + defstruct points: [], + bounds: nil, + metadata: %Metadata{} + + @doc """ + Builds a point cloud from a list of points. + + ## Arguments + + * `points` — list of `%Point{}` + * `opts`: + * `:compute_bounds` — boolean (default `true`) + * `:bounds` — explicit `%Bounds{}` (overrides compute) + * `:metadata` — `%Metadata{}` + + ## Returns + + `%PointCloud{}` + + ## Raises + + * `FunctionClauseError` if `points` is not a list, an element is not + point-like while bounds are computed, or `opts` is not a keyword list. + + ## Examples + + iex> alias ExCodecs.Spatial.{Point, PointCloud} + iex> cloud = PointCloud.new([Point.new(0, 0, 0), Point.new(1, 1, 1)]) + iex> PointCloud.size(cloud) + 2 + """ + @spec new([Point.t()], keyword()) :: t() + def new(points, opts \\ []) when is_list(points) do + compute_bounds? = Keyword.get(opts, :compute_bounds, true) + metadata = Keyword.get(opts, :metadata, %Metadata{}) + bounds = Keyword.get(opts, :bounds) + + %__MODULE__{ + points: points, + bounds: bounds || if(compute_bounds?, do: Bounds.from_points(points), else: nil), + metadata: metadata + } + end + + @doc """ + Number of points. + + ## Arguments + + * `cloud` — `%PointCloud{}` + + ## Returns + + non-negative integer + + ## Raises + + * `FunctionClauseError` if `cloud` is not a `%PointCloud{}`. + * `ArgumentError` if a manually constructed cloud has a non-list `:points`. + + ## Examples + + iex> ExCodecs.Spatial.PointCloud.size(ExCodecs.Spatial.PointCloud.new([])) + 0 + """ + @spec size(t()) :: non_neg_integer() + def size(%__MODULE__{points: points}), do: length(points) + + @doc """ + Returns `true` when every point has color (empty cloud → `false`). + + ## Arguments + + * `cloud` — `%PointCloud{}` + + ## Returns + + boolean + + ## Raises + + * `FunctionClauseError` if `cloud` is not a `%PointCloud{}`, or if any + point has an unsupported `:color` value. + + ## Examples + + iex> alias ExCodecs.Spatial.{Point, PointCloud} + iex> PointCloud.colored?(PointCloud.new([Point.new(0, 0, 0, color: {1, 2, 3})])) + true + """ + @spec colored?(t()) :: boolean() + def colored?(%__MODULE__{points: []}), do: false + def colored?(%__MODULE__{points: points}), do: Enum.all?(points, &Point.colored?/1) + + @doc """ + Returns `true` when every point has a normal (empty → `false`). + + ## Arguments + + * `cloud` — `%PointCloud{}` + + ## Returns + + boolean + + ## Raises + + * `FunctionClauseError` if `cloud` is not a `%PointCloud{}`, or if any + point has an unsupported `:normal` value. + + ## Examples + + iex> alias ExCodecs.Spatial.{Point, PointCloud} + iex> PointCloud.has_normals?(PointCloud.new([Point.new(0, 0, 0)])) + false + """ + @spec has_normals?(t()) :: boolean() + def has_normals?(%__MODULE__{points: []}), do: false + def has_normals?(%__MODULE__{points: points}), do: Enum.all?(points, &Point.has_normal?/1) + + @doc """ + Recomputes axis-aligned bounds from points. + + ## Arguments + + * `cloud` — `%PointCloud{}` + + ## Returns + + `%PointCloud{}` with updated `bounds` + + ## Raises + + * `FunctionClauseError` if `cloud` is not a `%PointCloud{}` or a point is + not a supported point-like value. + + ## Examples + + iex> alias ExCodecs.Spatial.{Point, PointCloud} + iex> cloud = PointCloud.new([Point.new(0, 0, 0)], compute_bounds: false) + iex> PointCloud.with_bounds(cloud).bounds != nil + true + """ + @spec with_bounds(t()) :: t() + def with_bounds(%__MODULE__{points: points} = cloud) do + %{cloud | bounds: Bounds.from_points(points)} + end + + @doc """ + Appends one point and expands bounds. + + Prefer `new/1` or `add_points/2` for bulk inserts (`++` is O(n)). + + ## Arguments + + * `cloud` — `%PointCloud{}` + * `point` — `%Point{}` + + ## Returns + + Updated `%PointCloud{}` + + ## Raises + + * `FunctionClauseError` unless the arguments are a `%PointCloud{}` and a + `%Point{}`, or if existing bounds are neither `nil` nor `%Bounds{}`. + + ## Examples + + iex> alias ExCodecs.Spatial.{Point, PointCloud} + iex> c = PointCloud.new([]) + iex> PointCloud.size(PointCloud.add_point(c, Point.new(1, 2, 3))) + 1 + """ + @spec add_point(t(), Point.t()) :: t() + def add_point(%__MODULE__{points: points, bounds: bounds} = cloud, %Point{} = point) do + new_bounds = + case bounds do + nil -> + Bounds.from_points([point]) + + %Bounds{} = b -> + {x, y, z} = Point.coords(point) + + %Bounds{ + min_x: min(b.min_x, x), + min_y: min(b.min_y, y), + min_z: min(b.min_z, z), + max_x: max(b.max_x, x), + max_y: max(b.max_y, y), + max_z: max(b.max_z, z) + } + end + + %{cloud | points: points ++ [point], bounds: new_bounds} + end + + @doc """ + Appends many points (via repeated `add_point/2`). + + ## Arguments + + * `cloud` — `%PointCloud{}` + * `new_points` — list of `%Point{}` + + ## Returns + + Updated `%PointCloud{}` + + ## Raises + + * `FunctionClauseError` unless `cloud` is a `%PointCloud{}` and + `new_points` is a list, or if any element is not a `%Point{}`. + + ## Examples + + iex> alias ExCodecs.Spatial.{Point, PointCloud} + iex> c = PointCloud.add_points(PointCloud.new([]), [Point.new(0, 0, 0), Point.new(1, 1, 1)]) + iex> PointCloud.size(c) + 2 + """ + @spec add_points(t(), [Point.t()]) :: t() + def add_points(%__MODULE__{} = cloud, new_points) when is_list(new_points) do + Enum.reduce(new_points, cloud, fn p, acc -> add_point(acc, p) end) + end +end diff --git a/lib/ex_codecs/spatial/stream.ex b/lib/ex_codecs/spatial/stream.ex new file mode 100644 index 0000000..ac35dec --- /dev/null +++ b/lib/ex_codecs/spatial/stream.ex @@ -0,0 +1,320 @@ +defmodule ExCodecs.Spatial.Stream do + @moduledoc """ + Stream helpers for spatial formats. + + **Important:** despite the name, most paths **materialize** the full source + (or full enumerable) then yield items. True incremental I/O for multi-GB + files is not implemented yet. + + Prefer `source: :file` when the argument is a filesystem path, or + `source: :binary` when it is an encoded payload. With `:auto` (default), a + binary is treated as a path only when it looks path-like (under 4 KiB, no + `ply`/`EXCP`/`GSPL` magic prefix, and contains `/` or `\\` or ends with + `.ply`/`.excp`/`.gspl`/`.bin`) **and** `File.regular?/1` is true. A real + file without separators/extensions is not auto-opened; a short slash-containing + binary that happens to be a regular path may be misread as a file. + """ + + alias ExCodecs.Error + alias ExCodecs.Spatial.Codec.{Binary, Gsplat, PLY} + alias ExCodecs.Spatial.{Gaussian, GaussianCloud, Point, PointCloud} + + @doc """ + Returns an enumerable over points or Gaussians decoded from a path or binary. + + The complete source and decoded cloud are currently materialized before + elements are emitted. + + ## Arguments + + * `source` (`Path.t() | binary()`) — a path or complete encoded payload. + Paths and payloads are both binaries, so `:source` controls resolution. + * `opts` (`keyword()`) — requires `:format`: `:ply`, + `:spatial_binary`, or `:gsplat`. `:source` may be `:auto` (default), + `:file`, or `:binary`; PLY also accepts `:as`. + + ## Returns + + An `Enumerable.t()` yielding `%Point{}` for PLY/EXCP point clouds or + `%Gaussian{}` for PLY/GSPL Gaussian clouds. A failure is delayed until + enumeration and represented by exactly one + `{:error, %ExCodecs.Error{}}` element: + + * `reason: :invalid_options` — `:format` is absent. + * `reason: :unsupported_codec` — `:format` is unknown. + * `reason: :io_error` — a selected file cannot be read. + * `reason: :invalid_data` — the payload is malformed, unsupported, or + truncated. + + ## Raises / exceptions + + A missing format does not raise. `Keyword.fetch/2` and decoder option access + raise `FunctionClauseError` when `opts` is not a proper keyword list. PLY + source dispatch raises `FunctionClauseError` for a non-binary `source`. + EXCP/GSPL source resolution has no clause for non-binaries and may return an + invalid value that causes `CaseClauseError`; an unsupported `:source` value + also raises `CaseClauseError`. Decoder exceptions may occur during + enumeration. + + ## Examples + + iex> alias ExCodecs.Spatial.{Point, PointCloud} + iex> {:ok, bin} = ExCodecs.Spatial.encode(PointCloud.new([Point.new(0.0, 0.0, 0.0)]), format: :ply) + iex> [%Point{}] = ExCodecs.Spatial.Stream.decode(bin, format: :ply) |> Enum.to_list() + iex> true + true + """ + @spec decode(Path.t() | binary(), keyword()) :: Enumerable.t() + def decode(source, opts \\ []) do + case Keyword.fetch(opts, :format) do + :error -> + error_stream( + Error.new(:invalid_options, + message: "Spatial stream_decode requires format: (e.g. format: :ply)" + ) + ) + + {:ok, format} -> + case format do + :ply -> + PLY.stream_decode(source, opts) + + :spatial_binary -> + stream_binary(source, opts) + + :gsplat -> + stream_gsplat(source, opts) + + other -> + error_stream( + Error.new(:unsupported_codec, + codec: other, + message: "Unsupported spatial stream format: #{inspect(other)}" + ) + ) + end + end + end + + @doc """ + Encodes an enumerable of points or Gaussians after collecting it in memory. + + A non-empty enumerable is classified by its first element. Empty input is + encoded as an empty `%PointCloud{}` for PLY/EXCP and as an empty + `%GaussianCloud{}` for GSPL. + + ## Arguments + + * `enumerable` (`Enumerable.t()`) — `%Point{}` or `%Gaussian{}` elements. + The list should be homogeneous and contain valid struct shapes. + * `opts` (`keyword()`) — requires `:format`: `:ply`, + `:spatial_binary`, or `:gsplat`. Remaining keys are forwarded to + `ExCodecs.Spatial.encode/2`. + + ## Returns + + * `{:ok, payload}` where `payload` is a `binary()`. + * `{:error, %ExCodecs.Error{reason: :invalid_options}}` if `:format` is + absent. + * `{:error, %ExCodecs.Error{reason: :unsupported_codec}}` for an unknown + format (including empty input). + * `{:error, %ExCodecs.Error{reason: :invalid_data}}` when the first item is + an error tuple or is not a `%Point{}`/`%Gaussian{}`, when cloud and format + are incompatible, or when codec validation fails. + + ## Raises / exceptions + + `Enum.to_list/1` raises `Protocol.UndefinedError` for a non-enumerable and + propagates exceptions raised by enumeration. Keyword access raises + `FunctionClauseError` for a non-keyword `opts`. Manually malformed point or + Gaussian structs can raise codec shape/numeric exceptions; PLY converts its + encoding exceptions to `:invalid_data`. + + ## Examples + + iex> alias ExCodecs.Spatial.Point + iex> {:ok, bin} = ExCodecs.Spatial.Stream.encode([Point.new(1.0, 0.0, 0.0)], format: :ply) + iex> is_binary(bin) + true + """ + @spec encode(Enumerable.t(), keyword()) :: {:ok, binary()} | {:error, Error.t()} + def encode(enumerable, opts \\ []) do + with {:ok, format} <- fetch_format(opts) do + do_encode(enumerable, format, opts) + end + end + + defp fetch_format(opts) do + case Keyword.fetch(opts, :format) do + {:ok, format} -> + {:ok, format} + + :error -> + {:error, + Error.new(:invalid_options, + message: "Spatial stream_encode requires format: (e.g. format: :ply)" + )} + end + end + + defp do_encode(enumerable, format, opts) do + items = Enum.to_list(enumerable) + + cond do + items == [] -> + encode_empty(format, opts) + + match?([%Point{} | _], items) -> + ExCodecs.Spatial.encode(PointCloud.new(items), opts) + + match?([%Gaussian{} | _], items) -> + ExCodecs.Spatial.encode(GaussianCloud.new(items), opts) + + match?([{:error, _} | _], items) -> + {:error, Error.new(:invalid_data, message: "Stream contained an error tuple")} + + true -> + {:error, + Error.new(:invalid_data, + message: "Stream encode expects Point or Gaussian structs" + )} + end + end + + @doc """ + Encodes spatial data and writes the binary to `path`. + + ## Arguments + + * `data` (`PointCloud.t() | GaussianCloud.t() | Enumerable.t()`) — a cloud, + or an enumerable of `%Point{}`/`%Gaussian{}` values. + * `path` (`Path.t()`) — destination accepted by `File.write/2`; parent + directories must already exist. + * `opts` (`keyword()`) — the options for `ExCodecs.Spatial.encode/2`, with + `:format` required for enumerable input and defaulting to `:ply` for an + already-built cloud. + + ## Returns + + * `:ok` after the complete payload is written. + * any `:invalid_options`, `:unsupported_codec`, or `:invalid_data` error + returned by encoding. + * `{:error, %ExCodecs.Error{reason: :io_error, details: reason}}` when + `File.write/2` returns a file/POSIX error such as `:enoent`, `:eacces`, + or `:enospc`. + + ## Raises / exceptions + + Normal `File.write/2` path/POSIX failures are returned. Invalid path terms or + non-path binaries can raise `FunctionClauseError` or `ArgumentError` in the + path/file APIs. Encoding also has the enumerable, keyword, and + malformed-struct exceptions documented by `encode/2`. + + ## Examples + + iex> alias ExCodecs.Spatial.{Point, PointCloud} + iex> path = Path.join(System.tmp_dir!(), "ex_codecs_doc_#{System.unique_integer([:positive])}.ply") + iex> :ok = ExCodecs.Spatial.Stream.encode_to_file(PointCloud.new([Point.new(0, 0, 0)]), path, format: :ply) + iex> {:ok, "ply" <> _} = File.read(path) + iex> File.rm!(path) + :ok + """ + @spec encode_to_file(Enumerable.t() | PointCloud.t() | GaussianCloud.t(), Path.t(), keyword()) :: + :ok | {:error, Error.t()} + def encode_to_file(data, path, opts \\ []) do + result = + case data do + %PointCloud{} -> ExCodecs.Spatial.encode(data, opts) + %GaussianCloud{} -> ExCodecs.Spatial.encode(data, opts) + enumerable -> encode(enumerable, opts) + end + + case result do + {:ok, binary} -> + case File.write(path, binary) do + :ok -> + :ok + + {:error, reason} -> + {:error, + Error.new(:io_error, + message: "Failed to write file: #{inspect(reason)}", + details: reason + )} + end + + {:error, _} = err -> + err + end + end + + defp encode_empty(:ply, opts), do: ExCodecs.Spatial.encode(PointCloud.new([]), opts) + defp encode_empty(:spatial_binary, opts), do: ExCodecs.Spatial.encode(PointCloud.new([]), opts) + defp encode_empty(:gsplat, opts), do: ExCodecs.Spatial.encode(GaussianCloud.new([]), opts) + + defp encode_empty(other, _) do + {:error, Error.new(:unsupported_codec, codec: other)} + end + + defp stream_binary(source, opts) do + case resolve_source(source, opts) do + {:ok, bin} -> Binary.stream_decode(bin, opts) + {:error, error} -> error_stream(error) + end + end + + defp stream_gsplat(source, opts) do + case resolve_source(source, opts) do + {:ok, bin} -> Gsplat.stream_decode(bin, opts) + {:error, error} -> error_stream(error) + end + end + + defp resolve_source(bin, opts) when is_binary(bin) do + case Keyword.get(opts, :source, :auto) do + :binary -> + {:ok, bin} + + :file -> + read_file(bin) + + :auto -> + if path_like?(bin) and File.regular?(bin) do + read_file(bin) + else + {:ok, bin} + end + end + end + + defp path_like?(bin) do + byte_size(bin) < 4096 and + not String.starts_with?(bin, "EXCP") and + not String.starts_with?(bin, "GSPL") and + not String.starts_with?(bin, "ply") and + (String.contains?(bin, "/") or String.contains?(bin, "\\") or + String.ends_with?(bin, [".excp", ".gspl", ".bin", ".ply"])) + end + + defp read_file(path) do + case File.read(path) do + {:ok, data} -> + {:ok, data} + + {:error, reason} -> + {:error, + Error.new(:io_error, message: "Failed to read file: #{inspect(reason)}", details: reason)} + end + end + + defp error_stream(error) do + Stream.resource( + fn -> {:error, error} end, + fn + {:error, e} -> {[{:error, e}], :done} + :done -> {:halt, :done} + end, + fn _ -> :ok end + ) + end +end diff --git a/lib/ex_codecs/spatial/transform.ex b/lib/ex_codecs/spatial/transform.ex new file mode 100644 index 0000000..df00efa --- /dev/null +++ b/lib/ex_codecs/spatial/transform.ex @@ -0,0 +1,118 @@ +defmodule ExCodecs.Spatial.Transform do + @moduledoc """ + Rigid/similarity transform metadata, not applied to points by this library. + + Codecs may embed transforms in headers when a format supports it. Application + of transforms is left to higher-level code. + + ## Fields + + * `:translation` — `{float(), float(), float()}` Cartesian offset; defaults + to `{0.0, 0.0, 0.0}`. Units match the associated spatial coordinates. + * `:rotation` — `{float(), float(), float(), float()}` scalar-first + `{w, x, y, z}` quaternion; defaults to identity + `{1.0, 0.0, 0.0, 0.0}`. Unit length is not enforced. + * `:scale` — `float()` uniform dimensionless scale; defaults to `1.0`. + + This module stores transform metadata but does not define composition order + or apply transforms; consumers must follow the enclosing format's convention. + + ## Example + + iex> ExCodecs.Spatial.Transform.new( + ...> translation: {10, 0, -2}, + ...> rotation: {1, 0, 0, 0}, + ...> scale: 0.5 + ...> ) + %ExCodecs.Spatial.Transform{ + translation: {10.0, 0.0, -2.0}, + rotation: {1.0, 0.0, 0.0, 0.0}, + scale: 0.5 + } + """ + + @typedoc """ + Transform metadata with translation, scalar-first quaternion rotation, and + uniform scale. See the module documentation for all fields, defaults, units + and conventions, and a construction example. + """ + @type t :: %__MODULE__{ + translation: {float(), float(), float()}, + rotation: {float(), float(), float(), float()}, + scale: float() + } + + defstruct translation: {0.0, 0.0, 0.0}, + rotation: {1.0, 0.0, 0.0, 0.0}, + scale: 1.0 + + @doc """ + Identity transform. + + ## Arguments + + None. + + ## Returns + + `%Transform{}` with zero translation, identity quaternion, scale `1.0`. + + ## Raises + + None. + + ## Examples + + iex> t = ExCodecs.Spatial.Transform.identity() + iex> t.scale + 1.0 + """ + @spec identity() :: t() + def identity, do: %__MODULE__{} + + @doc """ + Builds a transform. + + ## Arguments + + * `opts` — a keyword list with: + * `:translation` — numeric `{x, y, z}`; defaults to the origin and is + stored as floats. + * `:rotation` — numeric `{w, x, y, z}` scalar-first quaternion; defaults + to identity and is stored as floats. + * `:scale` — `number()`; defaults to `1.0` and is stored as a float. + + ## Returns + + `%Transform{}` + + ## Raises + + * `FunctionClauseError` for malformed or non-numeric translation/rotation + tuples, or if `opts` is not a keyword list. + * `ArithmeticError` if `:scale` is not numeric. + + ## Examples + + iex> t = ExCodecs.Spatial.Transform.new(translation: {1, 0, 0}, scale: 2) + iex> t.translation + {1.0, 0.0, 0.0} + """ + @spec new(keyword()) :: t() + def new(opts \\ []) do + %__MODULE__{ + translation: normalize_vec3(Keyword.get(opts, :translation, {0.0, 0.0, 0.0})), + rotation: normalize_quat(Keyword.get(opts, :rotation, {1.0, 0.0, 0.0, 0.0})), + scale: Keyword.get(opts, :scale, 1.0) * 1.0 + } + end + + defp normalize_vec3({x, y, z}) when is_number(x) and is_number(y) and is_number(z) do + {x * 1.0, y * 1.0, z * 1.0} + end + + defp normalize_quat({w, x, y, z}) + when is_number(w) and is_number(x) and is_number(y) and is_number(z) do + {w * 1.0, x * 1.0, y * 1.0, z * 1.0} + end +end diff --git a/livebooks/01_introduction.livemd b/livebooks/01_introduction.livemd index 7b9bb87..9172bf6 100644 --- a/livebooks/01_introduction.livemd +++ b/livebooks/01_introduction.livemd @@ -7,7 +7,7 @@ ex_codecs_dep = if File.exists?(local_path) do [{:ex_codecs, path: Path.join(__DIR__, "..")}, {:rustler, "~> 0.36"}] else - [{:ex_codecs, "~> 0.1"}] + [{:ex_codecs, "~> 0.2.0"}] end config = @@ -27,13 +27,20 @@ A **codec** (coder-decoder) is an abstraction for transforming data between two * **Encode** — transform data into a compact or structured form * **Decode** — recover the original data from the encoded form -For compression codecs, encoding means compressing and decoding means decompressing. But the codec abstraction extends beyond compression — it can represent hashing, checksums, binary encodings, and content-addressing transforms, all through the same `encode`/`decode` interface. +For compression codecs, encoding means compressing and decoding means +decompressing. ExCodecs is one framework with category APIs shaped for their +data: binary registry codecs use `ExCodecs.encode/3` / `decode/3`, while +point-cloud and Gaussian formats use `ExCodecs.Spatial`. ```elixir -# The universal codec interface: +# Binary registry interface: # {:ok, encoded} = ExCodecs.encode(:some_codec, data) # {:ok, decoded} = ExCodecs.decode(:some_codec, encoded) # decoded == data # round-trip guarantee +# +# Spatial category interface: +# {:ok, encoded} = ExCodecs.Spatial.encode(cloud, format: :ply) +# {:ok, decoded} = ExCodecs.Spatial.decode(encoded, format: :ply) ``` ## Why Codecs Matter @@ -54,9 +61,9 @@ A codec framework gives you a **single consistent API** over multiple algorithms ExCodecs is a **codec framework**, not just a compression library: -1. **Unified API** — `encode/3` and `decode/3` work identically across all codecs -2. **Runtime discovery** — query available codecs, check support, get metadata -3. **Extensible** — the `ExCodecs.Codec` behaviour lets you add new codec categories +1. **Specialized category APIs** — one framework without unsafe argument overloading +2. **Shared catalog** — query binary and spatial codecs, support, and metadata +3. **Extensible** — binary codecs use `ExCodecs.Codec`; domain categories can define suitable contracts 4. **NIF-native** — Rust-powered NIFs for production throughput 5. **Consistent errors** — structured `%ExCodecs.Error{}` for all failure modes @@ -104,8 +111,8 @@ IO.puts("Bzip2 block 9: #{byte_size(bz_max)} bytes") # Blosc2 shines with typed binary data data = :binary.copy(<<0, 0, 0, 0, 1, 1, 1, 1>>, 512) -{:ok, plain} = ExCodecs.encode(:blosc2, data, shuffle: :none) -{:ok, shuffled} = ExCodecs.encode(:blosc2, data, shuffle: :byte) +{:ok, plain} = ExCodecs.encode(:blosc2, data, typesize: 1, shuffle: :none) +{:ok, shuffled} = ExCodecs.encode(:blosc2, data, typesize: 1, shuffle: :byte) IO.puts("Original: #{byte_size(data)} bytes") IO.puts("Blosc2 (no shuffle): #{byte_size(plain)} bytes") @@ -116,7 +123,9 @@ IO.puts("Blosc2 (byte shuffle): #{byte_size(shuffled)} bytes") ```elixir codecs = ExCodecs.available_codecs() -IO.puts("Available codecs: #{inspect(codecs)}") +IO.puts("Shared catalog: #{inspect(codecs)}") +IO.puts("Compression: #{inspect(ExCodecs.available_codecs(:compression))}") +IO.puts("Spatial: #{inspect(ExCodecs.available_codecs(:spatial))}") ``` ### Codec Details @@ -127,6 +136,7 @@ for codec <- codecs do IO.puts(String.duplicate("-", 50)) IO.puts("Codec: #{info.name}") IO.puts("Category: #{info.category}") + IO.puts("Interface: #{info.interface}") IO.puts("Native?: #{info.native?}") IO.puts("Streaming?: #{info.streaming?}") IO.puts("Configurable?: #{info.configurable?}") @@ -138,11 +148,12 @@ end | Codec | Category | Configurable | Streaming | Best For | | --------- | ----------- | ---------------------- | --------- | ------------------------------ | -| `:zstd` | compression | Yes (level 1–22) | Yes | General-purpose, high ratio | -| `:lz4` | compression | Yes (level 1–16) | No | Real-time, low latency | +| `:zstd` | compression | Yes (level 1–22) | No | General-purpose, high ratio | +| `:lz4` | compression | No | No | Real-time, low latency | | `:snappy` | compression | No | No | Short-lived data, low overhead | | `:bzip2` | compression | Yes (block_size 1–9) | No | Archival, maximum ratio | -| `:blosc2` | compression | Yes (many options) | Yes | Numerical/array data | +| `:blosc2` | compression | Yes (codec/shuffle) | No | Numerical/array data | +| `:ply` / `:spatial_binary` / `:gsplat` | spatial | Format-specific | No | Point clouds / Gaussians | ## Error Handling diff --git a/livebooks/02_compression_fundamentals.livemd b/livebooks/02_compression_fundamentals.livemd index 016262d..60aa9ac 100644 --- a/livebooks/02_compression_fundamentals.livemd +++ b/livebooks/02_compression_fundamentals.livemd @@ -7,7 +7,7 @@ ex_codecs_dep = if File.exists?(local_path) do [{:ex_codecs, path: Path.join(__DIR__, "..")}, {:rustler, "~> 0.36"}] else - [{:ex_codecs, "~> 0.1"}] + [{:ex_codecs, "~> 0.2.0"}] end config = @@ -55,7 +55,11 @@ ExCodecs provides **lossless** codecs — decoded data is bit-for-bit identical ```elixir data = :crypto.strong_rand_bytes(8192) -for codec <- ExCodecs.available_codecs() do +compression_codecs = + ExCodecs.Compression.available_codecs() + |> Enum.map(& &1.name) + +for codec <- compression_codecs do {:ok, enc} = ExCodecs.encode(codec, data) {:ok, dec} = ExCodecs.decode(codec, enc) IO.puts("#{String.pad_trailing(inspect(codec), 10)} lossless: #{dec == data}") @@ -102,9 +106,9 @@ Blosc2 reorders bytes to create longer runs before applying an internal compress # Numerical data with patterns across float64 values floats = for i <- 1..2048, into: <<>>, do: <> -{:ok, blosc_none} = ExCodecs.encode(:blosc2, floats, shuffle: :none) -{:ok, blosc_byte} = ExCodecs.encode(:blosc2, floats, shuffle: :byte) -{:ok, blosc_bit} = ExCodecs.encode(:blosc2, floats, shuffle: :bit) +{:ok, blosc_none} = ExCodecs.encode(:blosc2, floats, typesize: 8, shuffle: :none) +{:ok, blosc_byte} = ExCodecs.encode(:blosc2, floats, typesize: 8, shuffle: :byte) +{:ok, blosc_bit} = ExCodecs.encode(:blosc2, floats, typesize: 8, shuffle: :bit) {:ok, zstd_plain} = ExCodecs.encode(:zstd, floats) IO.puts("Original: #{byte_size(floats)} bytes") diff --git a/livebooks/03_codec_comparison.livemd b/livebooks/03_codec_comparison.livemd index b5b372b..f3d3468 100644 --- a/livebooks/03_codec_comparison.livemd +++ b/livebooks/03_codec_comparison.livemd @@ -7,7 +7,7 @@ ex_codecs_dep = if File.exists?(local_path) do [{:ex_codecs, path: Path.join(__DIR__, "..")}, {:rustler, "~> 0.36"}] else - [{:ex_codecs, "~> 0.1"}] + [{:ex_codecs, "~> 0.2.0"}] end config = @@ -59,7 +59,10 @@ end ```elixir compression_results = for {dname, data} <- datasets, codec <- codecs do - opts = if codec == :blosc2, do: [cname: :zstd, clevel: 5, shuffle: :byte], else: [] + opts = + if codec == :blosc2, + do: [cname: :zstd, clevel: 5, shuffle: :byte, typesize: 8], + else: [] {:ok, enc} = ExCodecs.encode(codec, data, opts) %{ dataset: dname, @@ -104,7 +107,10 @@ VegaLite.new(width: 700, height: 350) iterations = 20 speed_results = for {dname, data} <- datasets, codec <- codecs do - opts = if codec == :blosc2, do: [cname: :zstd, clevel: 5, shuffle: :byte], else: [] + opts = + if codec == :blosc2, + do: [cname: :zstd, clevel: 5, shuffle: :byte, typesize: 8], + else: [] {:ok, enc} = ExCodecs.encode(codec, data, opts) {enc_time, _} = :timer.tc(fn -> @@ -159,7 +165,10 @@ VegaLite.new(width: 700, height: 350) ```elixir memory_results = for codec <- codecs do - opts = if codec == :blosc2, do: [cname: :zstd, clevel: 5, shuffle: :byte], else: [] + opts = + if codec == :blosc2, + do: [cname: :zstd, clevel: 5, shuffle: :byte, typesize: 8], + else: [] {:ok, info} = ExCodecs.codec_info(codec) mem_before = Process.info(self(), :heap_size) |> elem(1) @@ -238,7 +247,7 @@ recommendation = case {use_case_val, data_type_val} do {:tiny, _} -> {:snappy, "Low overhead even on very small payloads. No configuration needed."} {:ratio, :array} -> {:blosc2, "Shuffle+compress gives best ratios on typed arrays."} {:ratio, _} -> {:bzip2, "Highest compression ratio for general data. Slow but compact."} - {:numeric, _} -> {:blosc2, "Purpose-built for numerical data with shuffle filters and threading."} + {:numeric, _} -> {:blosc2, "Purpose-built for numerical data with shuffle filters."} {:balanced, :array} -> {:blosc2, "Good ratio on typed data with decent speed."} {:balanced, _} -> {:zstd, "Best all-around codec. Configurable from fast (level 1) to compact (level 22)."} end @@ -253,9 +262,9 @@ IO.puts("Streaming: #{info.streaming?}") default_opts = case codec do :zstd -> [level: 3] - :lz4 -> [level: 1] + :lz4 -> [] :bzip2 -> [block_size: 9] - :blosc2 -> [cname: :zstd, clevel: 5, shuffle: :byte] + :blosc2 -> [cname: :zstd, clevel: 5, shuffle: :byte, typesize: 8] :snappy -> [] end IO.puts("Suggested options: #{inspect(default_opts)}") @@ -293,11 +302,11 @@ IO.puts(flowchart) | ------------ | ------------ | ---------- | ------------ | ----------- | ------------- | | Speed | Very Fast | Very Fast | Fast | Slow | Medium | | Ratio | Low | Low | High | Very High | High (arrays) | -| Configurable | Level 1–16 | No | Level 1–22 | Block 1–9 | Many options | -| Streaming | No | No | Yes | No | Yes | +| Configurable | Fixed profile | No | Level 1–22 | Block 1–9 | Codec/shuffle | +| Streaming | No | No | No | No | No | | Best For | Hot paths | Short data | General | Archival | Arrays | | Shuffle | — | — | — | — | Byte/Bit | -| Multi-thread | No | No | No | No | Yes | +| Multi-thread | No | No | No | No | No | ## Next Steps diff --git a/livebooks/04_building_storage_systems.livemd b/livebooks/04_building_storage_systems.livemd index 72974f5..ed9877f 100644 --- a/livebooks/04_building_storage_systems.livemd +++ b/livebooks/04_building_storage_systems.livemd @@ -7,7 +7,7 @@ ex_codecs_dep = if File.exists?(local_path) do [{:ex_codecs, path: Path.join(__DIR__, "..")}, {:rustler, "~> 0.36"}] else - [{:ex_codecs, "~> 0.1"}] + [{:ex_codecs, "~> 0.2.0"}] end config = @@ -124,7 +124,7 @@ end ### Using the Compressed Store ```elixir - {:ok, _pid} = CompressedStore.start_link(codec: :zstd, level: 3) +{:ok, _pid} = CompressedStore.start_link(codec: :zstd, opts: [level: 3]) CompressedStore.put(:doc1, String.duplicate("Hello, World! ", 1000)) CompressedStore.put(:doc2, :crypto.strong_rand_bytes(4096) |> Base.encode64()) @@ -308,6 +308,12 @@ case ExCodecs.encode(:zstd, "some data") do {:error, %ExCodecs.Error{reason: :unsupported_codec}} -> {:error, :codec_missing} + {:error, %ExCodecs.Error{reason: :codec_unavailable}} -> + {:error, :codec_unavailable} + + {:error, %ExCodecs.Error{reason: :nif_not_loaded}} -> + {:error, :nif_not_loaded} + {:error, %ExCodecs.Error{reason: :invalid_data} = err} -> IO.puts("Invalid data: #{err.message}") {:error, :bad_input} @@ -315,9 +321,23 @@ case ExCodecs.encode(:zstd, "some data") do {:error, %ExCodecs.Error{reason: :compression_failed} = err} -> IO.puts("Compression failed: #{err.message}") {:error, :compress_failed} + + {:error, %ExCodecs.Error{} = err} -> + IO.puts("Structured error #{err.reason}: #{err.message}") + {:error, err.reason} end ``` +```elixir +# Bound decompression for untrusted payloads (default is 256 MiB) +{:ok, compressed} = ExCodecs.encode(:zstd, String.duplicate("x", 1024)) + +{:error, %ExCodecs.Error{reason: :output_limit_exceeded}} = + ExCodecs.decode(:zstd, compressed, max_output_size: 16) + +{:ok, _decoded} = ExCodecs.decode(:zstd, compressed, max_output_size: 4096) +``` + ```elixir defmodule SafeCompress do def call(codec, data, opts \\ []) do diff --git a/livebooks/05_zarr_style_workloads.livemd b/livebooks/05_zarr_style_workloads.livemd index 3c13813..8110579 100644 --- a/livebooks/05_zarr_style_workloads.livemd +++ b/livebooks/05_zarr_style_workloads.livemd @@ -7,7 +7,7 @@ ex_codecs_dep = if File.exists?(local_path) do [{:ex_codecs, path: Path.join(__DIR__, "..")}, {:rustler, "~> 0.36"}] else - [{:ex_codecs, "~> 0.1"}] + [{:ex_codecs, "~> 0.2.0"}] end config = @@ -75,11 +75,11 @@ The shuffle filter is the key to Blosc2's effectiveness on array data: data = :binary.copy(<<1.0::float-64>>, 50_000) <> :binary.copy(<<2.0::float-64>>, 50_000) # No shuffle -{:ok, c_none} = ExCodecs.encode(:blosc2, data, shuffle: :none) +{:ok, c_none} = ExCodecs.encode(:blosc2, data, typesize: 8, shuffle: :none) # Byte shuffle - reorders bytes for better compression -{:ok, c_byte} = ExCodecs.encode(:blosc2, data, shuffle: :byte) +{:ok, c_byte} = ExCodecs.encode(:blosc2, data, typesize: 8, shuffle: :byte) # Bit shuffle - reorders bits for even better compression on some data -{:ok, c_bit} = ExCodecs.encode(:blosc2, data, shuffle: :bit) +{:ok, c_bit} = ExCodecs.encode(:blosc2, data, typesize: 8, shuffle: :bit) IO.puts("No shuffle: #{byte_size(c_none)} bytes") IO.puts("Byte shuffle: #{byte_size(c_byte)} bytes") diff --git a/livebooks/06_spatial_codecs.livemd b/livebooks/06_spatial_codecs.livemd new file mode 100644 index 0000000..9ebfdf8 --- /dev/null +++ b/livebooks/06_spatial_codecs.livemd @@ -0,0 +1,210 @@ +# Spatial Codecs: Point Clouds and Gaussian Splats + +```elixir +local_path = Path.join(__DIR__, "../mix.exs") + +ex_codecs_dep = + if File.exists?(local_path) do + [{:ex_codecs, path: Path.join(__DIR__, "..")}, {:rustler, "~> 0.36"}] + else + [{:ex_codecs, "~> 0.2.0"}] + end + +config = + if File.exists?(local_path) do + [rustler_precompiled: [force_build: [ex_codecs: true]]] + else + [] + end + +Mix.install(ex_codecs_dep, config: config) +``` + +## One Framework, Specialized Category APIs + +Compression codecs and spatial formats share discovery metadata and +`%ExCodecs.Error{}` results. Their operation APIs differ because compression +maps binary to binary, while spatial codecs map domain structs to formats. + +```elixir +ExCodecs.available_codecs(:spatial) +``` + +```elixir +for format <- ExCodecs.Spatial.available_formats() do + {:ok, info} = ExCodecs.codec_info(format) + + %{ + format: info.name, + interface: info.interface, + configurable?: info.configurable?, + version: info.version + } +end +``` + +## Build a Point Cloud + +`Point.new/4` stores coordinates as floats and normalizes attribute keys to +strings. `PointCloud.new/2` computes axis-aligned bounds by default. + +```elixir +alias ExCodecs.Spatial.{Gaussian, GaussianCloud, Point, PointCloud} + +points = [ + Point.new(0, 0, 0, + color: {255, 32, 32}, + normal: {0.0, 0.0, 1.0}, + attributes: %{intensity: 0.25} + ), + Point.new(1, 0, 0, + color: {32, 255, 32}, + normal: {0.0, 0.0, 1.0}, + attributes: %{intensity: 0.5} + ), + Point.new(0, 1, 0, + color: {32, 32, 255}, + normal: {0.0, 0.0, 1.0}, + attributes: %{intensity: 0.75} + ) +] + +cloud = PointCloud.new(points) + +%{ + points: PointCloud.size(cloud), + bounds: cloud.bounds, + attribute_keys: cloud.points |> hd() |> Map.fetch!(:attributes) |> Map.keys() +} +``` + +## PLY Interchange + +PLY supports ASCII and binary little-/big-endian representations. The default +is binary little-endian. + +```elixir +{:ok, ply_ascii} = + ExCodecs.Spatial.encode(cloud, + format: :ply, + ply_format: :ascii, + comments: ["created by the ExCodecs spatial Livebook"] + ) + +ply_ascii +|> String.split("\n") +|> Enum.take(14) +|> Enum.join("\n") +``` + +```elixir +{:ok, ply_binary} = ExCodecs.Spatial.encode(cloud, format: :ply) +{:ok, decoded_ply} = ExCodecs.Spatial.decode(ply_binary, format: :ply) + +%{ + encoded_bytes: byte_size(ply_binary), + decoded_points: PointCloud.size(decoded_ply), + first_point: hd(decoded_ply.points) +} +``` + +## Compact Point-Cloud Binary + +`:spatial_binary` uses the versioned ExCodecs point format (`EXCP`). It is +intended for compact ExCodecs-to-ExCodecs interchange. + +```elixir +{:ok, excp} = ExCodecs.Spatial.encode(cloud, format: :spatial_binary) +{:ok, decoded_excp} = ExCodecs.Spatial.decode(excp, format: :spatial_binary) + +%{ + magic: binary_part(excp, 0, 4), + encoded_bytes: byte_size(excp), + decoded_points: PointCloud.size(decoded_excp) +} +``` + +## Spatial Encoding Plus Compression + +Spatial encoding chooses a geometry format. Compression is a separate, +composable binary step. + +```elixir +{:ok, compressed_excp} = ExCodecs.encode(:zstd, excp, level: 7) +{:ok, restored_excp} = ExCodecs.decode(:zstd, compressed_excp) +{:ok, restored_cloud} = + ExCodecs.Spatial.decode(restored_excp, format: :spatial_binary) + +%{ + excp_bytes: byte_size(excp), + compressed_bytes: byte_size(compressed_excp), + round_trip_points: PointCloud.size(restored_cloud) +} +``` + +## Gaussian Splats + +`Gaussian` uses scalar-first quaternions (`{w, x, y, z}`). `:gsplat` is the +versioned ExCodecs Gaussian format (`GSPL`). + +```elixir +gaussian_cloud = + GaussianCloud.new([ + Gaussian.new({0, 0, 0}, + scale: {0.1, 0.2, 0.1}, + opacity: 0.9, + color: {1.0, 0.2, 0.1} + ), + Gaussian.new({1, 0.5, -0.25}, + rotation: {0.9239, 0.0, 0.3827, 0.0}, + scale: {0.25, 0.1, 0.15}, + opacity: 0.65, + color: {0.1, 0.4, 1.0} + ) + ]) + +{:ok, gspl} = ExCodecs.Spatial.encode(gaussian_cloud, format: :gsplat) +{:ok, decoded_gaussians} = ExCodecs.Spatial.decode(gspl, format: :gsplat) + +%{ + magic: binary_part(gspl, 0, 4), + encoded_bytes: byte_size(gspl), + decoded_gaussians: GaussianCloud.size(decoded_gaussians) +} +``` + +## Enumerable Helpers + +The current `stream_encode/2` and `stream_decode/2` names describe an +enumerable-facing API. In v0.2.0 they still collect the complete enumerable or +payload in memory; they are not incremental file I/O. + +```elixir +{:ok, streamed_payload} = + ExCodecs.Spatial.stream_encode(points, format: :spatial_binary) + +streamed_points = + streamed_payload + |> ExCodecs.Spatial.stream_decode(format: :spatial_binary, source: :binary) + |> Enum.to_list() + +length(streamed_points) +``` + +## Category-Safe Dispatch + +The shared catalog does not overload the binary API with struct inputs. +Choosing a spatial catalog entry through `ExCodecs.encode/3` returns guidance +to use the spatial category API. + +```elixir +{:error, error} = ExCodecs.encode(:ply, ply_binary) + +%{reason: error.reason, message: error.message} +``` + +## Next Steps + +* Read **Understanding Spatial Codecs** for schema and format trade-offs. +* Read **Spatial Wire Formats** for the frozen EXCP and GSPL layouts. +* Add Zstd after spatial encoding when storage or transfer size matters. diff --git a/mix.exs b/mix.exs index 255a09a..8989c44 100644 --- a/mix.exs +++ b/mix.exs @@ -1,7 +1,7 @@ defmodule ExCodecs.MixProject do use Mix.Project - @version "0.1.1" + @version "0.2.0" @source_url "https://github.com/thanos/codecs" def project do @@ -19,7 +19,7 @@ defmodule ExCodecs.MixProject do test_coverage: [ tool: ExCoveralls, ignore_modules: [ExCodecs.Native], - threshold: 90 + threshold: 95 ], preferred_cli_env: [ coveralls: :test, @@ -71,7 +71,8 @@ defmodule ExCodecs.MixProject do defp package do [ - description: "An extensible BEAM-native codec framework for Elixir", + description: + "An extensible BEAM-native codec framework for Elixir — compression and spatial formats", licenses: ["Apache-2.0"], links: %{ "GitHub" => @source_url, @@ -84,6 +85,9 @@ defmodule ExCodecs.MixProject do "native/ex_codecs_native/Cargo.toml", "native/ex_codecs_native/Cargo.lock", "priv", + "guides", + "livebooks", + "docs", "checksum-*.exs", "mix.exs", "README.md", @@ -97,7 +101,15 @@ defmodule ExCodecs.MixProject do main: "ExCodecs", source_url: @source_url, source_ref: "v#{@version}", - extras: Path.wildcard("guides/**/*.md") ++ Path.wildcard("livebooks/**/*.livemd") + extras: + Path.wildcard("guides/**/*.md") ++ + Path.wildcard("livebooks/**/*.livemd") ++ + ["docs/spatial_formats.md"], + groups_for_extras: [ + Guides: ~r"guides/", + Livebooks: ~r"livebooks/", + "Architecture & formats": ~r"docs/" + ] ] end diff --git a/native/ex_codecs_native/Cargo.lock b/native/ex_codecs_native/Cargo.lock new file mode 100644 index 0000000..cba4fec --- /dev/null +++ b/native/ex_codecs_native/Cargo.lock @@ -0,0 +1,302 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "adler2" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" + +[[package]] +name = "blosc2-pure-rs" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8c480a5f547dc795da409a61d030108755f0b2ca67108054fc7abb7b4e8c0fe" +dependencies = [ + "flate2", + "lz4-pure-rs", + "rayon", + "zstd-pure-rs", +] + +[[package]] +name = "bzip2" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f3a53fac24f34a81bc9954b5d6cfce0c21e18ec6959f44f56e8e90e4bb7c346c" +dependencies = [ + "libbz2-rs-sys", +] + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "crc32fast" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "crossbeam-deque" +version = "0.8.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5181e0de7b61eb03a81e347d6dd8797bae9da5146707b51077e2d71a54ec0ceb" +dependencies = [ + "crossbeam-epoch", + "crossbeam-utils", +] + +[[package]] +name = "crossbeam-epoch" +version = "0.9.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2d6914041f254d6e9176c01941b21115dcfb7089e55135a35411081bd106ef3f" +dependencies = [ + "crossbeam-utils", +] + +[[package]] +name = "crossbeam-utils" +version = "0.8.22" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "61803da095bee82a81bb1a452ecc25d3b2f1416d1897eb86430c6159ef717c17" + +[[package]] +name = "either" +version = "1.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e" + +[[package]] +name = "ex_codecs_native" +version = "0.2.0" +dependencies = [ + "blosc2-pure-rs", + "bzip2", + "flate2", + "lz4_flex", + "rustler", + "snap", + "structured-zstd", +] + +[[package]] +name = "flate2" +version = "1.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c" +dependencies = [ + "crc32fast", + "miniz_oxide", + "zlib-rs", +] + +[[package]] +name = "heck" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" + +[[package]] +name = "inventory" +version = "0.3.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4f0c30c76f2f4ccee3fe55a2435f691ca00c0e4bd87abe4f4a851b1d4dac39b" +dependencies = [ + "rustversion", +] + +[[package]] +name = "libbz2-rs-sys" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "34b357333733e8260735ba5894eb928c02ecc69c78715f01a8019e7fa7f2db4c" + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "libloading" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55" +dependencies = [ + "cfg-if", + "windows-link", +] + +[[package]] +name = "lz4-pure-rs" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c9999337316af402052119585343e87a43d4d549b08cf51f53670a187c7c153a" +dependencies = [ + "libc", +] + +[[package]] +name = "lz4_flex" +version = "0.11.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "373f5eceeeab7925e0c1098212f2fbc4d416adec9d35051a6ab251e824c1854a" +dependencies = [ + "twox-hash", +] + +[[package]] +name = "miniz_oxide" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316" +dependencies = [ + "adler2", + "simd-adler32", +] + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "rayon" +version = "1.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fb39b166781f92d482534ef4b4b1b2568f42613b53e5b6c160e24cfbfa30926d" +dependencies = [ + "either", + "rayon-core", +] + +[[package]] +name = "rayon-core" +version = "1.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "22e18b0f0062d30d4230b2e85ff77fdfe4326feb054b9783a3460d8435c8ab91" +dependencies = [ + "crossbeam-deque", + "crossbeam-utils", +] + +[[package]] +name = "regex-lite" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cab834c73d247e67f4fae452806d17d3c7501756d98c8808d7c9c7aa7d18f973" + +[[package]] +name = "rustler" +version = "0.36.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3fe55230a9c379733dd38ee67d4072fa5c558b2e22b76b0e7f924390456e003" +dependencies = [ + "inventory", + "libloading", + "regex-lite", + "rustler_codegen", +] + +[[package]] +name = "rustler_codegen" +version = "0.36.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eb3b8de901ae61418e2036245d28e41ef58080d04f40b68430471ae36a4e84ed" +dependencies = [ + "heck", + "inventory", + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "rustversion" +version = "1.0.22" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d" + +[[package]] +name = "simd-adler32" +version = "0.3.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a219298ac11a56ea9a6d2120044824d6f01aeb034955e7af7bc16858527deea" + +[[package]] +name = "snap" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b6b67fb9a61334225b5b790716f609cd58395f895b3fe8b328786812a40bc3b" + +[[package]] +name = "structured-zstd" +version = "0.0.48" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c744bb12e451d61c0e22a4db2728551bc638441db85b417dafbaf91041deb398" +dependencies = [ + "twox-hash", +] + +[[package]] +name = "syn" +version = "2.0.117" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "twox-hash" +version = "2.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ea3136b675547379c4bd395ca6b938e5ad3c3d20fad76e7fe85f9e0d011419c" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "zlib-rs" +version = "0.6.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b142a20ec14a91d5bc708c1dc21b080c550113d8aa77afa29635673a65dd02c5" + +[[package]] +name = "zstd-pure-rs" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4f5e05972e1a97c5261f46a9bb305fc2076171c3cf42033d4bf0b6c1942f7f8a" diff --git a/native/ex_codecs_native/Cargo.toml b/native/ex_codecs_native/Cargo.toml index fe86b56..48a44f1 100644 --- a/native/ex_codecs_native/Cargo.toml +++ b/native/ex_codecs_native/Cargo.toml @@ -1,9 +1,11 @@ [package] name = "ex_codecs_native" -version = "0.1.0" +version = "0.2.0" authors = ["ExCodecs Team"] edition = "2021" -rust-version = "1.85" +# structured-zstd declares MSRV 1.92 but calls the now-safe __cpuid without +# an unsafe block on x86_64, which requires rustc >= 1.94 (E0133 on older). +rust-version = "1.94" [lib] name = "ex_codecs_native" @@ -12,10 +14,15 @@ crate-type = ["cdylib"] [dependencies] rustler = { version = "0.36", default-features = false, features = ["derive"] } -zstd = "0.13" +# Pure-Rust codecs only — no C compression toolchain (no cmake/c-blosc2). +structured-zstd = "0.0.48" lz4_flex = "0.11" snap = "1.1" -bzip2 = "0.4" +flate2 = { version = "1", default-features = false, features = ["rust_backend"] } +# bzip2 0.6 defaults to pure-Rust libbz2-rs-sys (no C libbzip2). +bzip2 = "0.6" +# Official Blosc2 chunk format (BloscLZ, LZ4, LZ4HC, Zlib, Zstd + filters). +blosc2-pure-rs = { version = "0.2.4", default-features = false, features = ["zlib-rs"] } [features] default = ["nif_version_2_17"] @@ -25,4 +32,4 @@ nif_version_2_17 = ["rustler/nif_version_2_17"] opt-level = 3 lto = true codegen-units = 1 -strip = true \ No newline at end of file +strip = true diff --git a/native/ex_codecs_native/src/atoms.rs b/native/ex_codecs_native/src/atoms.rs index c573288..68c9caf 100644 --- a/native/ex_codecs_native/src/atoms.rs +++ b/native/ex_codecs_native/src/atoms.rs @@ -10,4 +10,5 @@ atoms! { compression_failed, decompression_failed, nif_not_loaded, -} \ No newline at end of file + output_limit_exceeded, +} diff --git a/native/ex_codecs_native/src/blosc2_codec.rs b/native/ex_codecs_native/src/blosc2_codec.rs index 52deb90..a44bfbd 100644 --- a/native/ex_codecs_native/src/blosc2_codec.rs +++ b/native/ex_codecs_native/src/blosc2_codec.rs @@ -1,182 +1,42 @@ -use rustler::{Binary, Encoder, Env, Term}; +//! C-Blosc2-compatible chunk compress/decompress (pure Rust). +//! +//! Uses [`blosc2_pure_rs`] for the official chunk wire format (header, blocks, +//! filters, codecs including BloscLZ and LZ4HC). No C-Blosc2 is linked. -use crate::atoms; -use crate::util::encode_binary; - -pub fn version() -> String { - "2.x-pure-rust".to_string() -} - -const BLOSC2_MAGIC: u8 = 0x2c; -const BLOSC2_VERSION: u8 = 2; -const BLOSC_MIN_HEADER_LENGTH: usize = 16; - -const BLOSC_BLOSCLZ: u8 = 0; -const BLOSC_LZ4: u8 = 1; -const BLOSC_LZ4HC: u8 = 2; -const BLOSC_SNAPPY: u8 = 3; -const BLOSC_ZLIB: u8 = 4; -const BLOSC_ZSTD: u8 = 5; - -const BLOSC_NOSHUFFLE: u8 = 0; -const BLOSC_BYTESHUFFLE: u8 = 1; -const BLOSC_BITSHUFFLE: u8 = 2; - -fn byte_shuffle(data: &[u8], typesize: usize) -> Vec { - if typesize <= 1 || data.len() < typesize { - return data.to_vec(); - } - - let n_elements = data.len() / typesize; - let mut result = vec![0u8; data.len()]; - - for i in 0..n_elements { - for j in 0..typesize { - result[j * n_elements + i] = data[i * typesize + j]; - } - } - - let offset = n_elements * typesize; - if offset < data.len() { - result[offset..].copy_from_slice(&data[offset..]); - } - - result -} - -fn byte_unshuffle(data: &[u8], typesize: usize) -> Vec { - if typesize <= 1 || data.len() < typesize { - return data.to_vec(); - } - - let n_elements = data.len() / typesize; - let mut result = vec![0u8; data.len()]; - - for i in 0..n_elements { - for j in 0..typesize { - result[i * typesize + j] = data[j * n_elements + i]; - } - } - - let offset = n_elements * typesize; - if offset < data.len() { - result[offset..].copy_from_slice(&data[offset..]); - } - - result -} - -fn bit_shuffle(data: &[u8], typesize: usize) -> Vec { - if typesize == 0 || data.is_empty() { - return data.to_vec(); - } - - let n_elements = data.len() / typesize; - if n_elements == 0 { - return data.to_vec(); - } - - let n_full_blocks = n_elements / 8; - let block_bytes = n_full_blocks * 8 * typesize; - let mut result = vec![0u8; data.len()]; - let mut out_pos = 0; - - for block in 0..n_full_blocks { - let block_start = block * 8; - for byte_idx in 0..typesize { - for bit_idx in 0..8 { - let mut acc: u8 = 0; - for i in 0..8 { - let src = data[(block_start + i) * typesize + byte_idx]; - if src & (1 << (7 - bit_idx)) != 0 { - acc |= 1 << (7 - i); - } - } - result[out_pos] = acc; - out_pos += 1; - } - } - } - - if block_bytes < data.len() { - result[block_bytes..data.len()].copy_from_slice(&data[block_bytes..data.len()]); - } +use rustler::{Binary, Env, Term}; - result -} - -fn bit_unshuffle(data: &[u8], typesize: usize) -> Vec { - if typesize == 0 || data.is_empty() { - return data.to_vec(); - } - - let n_elements = data.len() / typesize; - if n_elements == 0 { - return data.to_vec(); - } +use crate::atoms; +use crate::util::{err, ok_binary}; - let n_full_blocks = n_elements / 8; - let block_bytes = n_full_blocks * 8 * typesize; - let mut result = vec![0u8; data.len()]; - let mut in_pos = 0; +use blosc2_pure_rs::{ + blosc2_compress_ctx, blosc2_create_cctx, blosc2_create_dctx, blosc2_decompress_ctx, + blosc2_free_ctx, blosc2_get_version_string, CParams, DParams, BLOSC2_MAX_OVERHEAD, + BLOSC_BITSHUFFLE, BLOSC_BLOSCLZ, BLOSC_LZ4, BLOSC_LZ4HC, BLOSC_NOSHUFFLE, BLOSC_SHUFFLE, + BLOSC_ZLIB, BLOSC_ZSTD, +}; - for block in 0..n_full_blocks { - let block_start = block * 8; - for byte_idx in 0..typesize { - let mut transposed = [0u8; 8]; - for k in 0..8 { - transposed[k] = data[in_pos]; - in_pos += 1; - } - for i in 0..8 { - let mut byte_val: u8 = 0; - for bit_idx in 0..8 { - if transposed[bit_idx] & (1 << (7 - i)) != 0 { - byte_val |= 1 << (7 - bit_idx); - } - } - result[(block_start + i) * typesize + byte_idx] = byte_val; - } - } - } - - if block_bytes < data.len() { - result[block_bytes..data.len()].copy_from_slice(&data[block_bytes..data.len()]); - } - - result +pub fn version() -> String { + format!("c-blosc2-chunk/{}", blosc2_get_version_string()) } -fn internal_compress(data: &[u8], cname: u8, clevel: u8) -> Result, ()> { - match cname { - BLOSC_BLOSCLZ | BLOSC_LZ4 | BLOSC_LZ4HC => Ok(lz4_flex::compress_prepend_size(data)), - BLOSC_SNAPPY => { - let mut encoder = snap::raw::Encoder::new(); - encoder.compress_vec(data).map_err(|_| ()) - } - BLOSC_ZSTD => { - let level = (clevel as i32).clamp(1, 22); - zstd::bulk::compress(data, level).map_err(|_| ()) - } - BLOSC_ZLIB => Ok(lz4_flex::compress_prepend_size(data)), - _ => Err(()), +fn filter_from_shuffle(shuffle: u8) -> u8 { + match shuffle { + 0 => BLOSC_NOSHUFFLE, + 1 => BLOSC_SHUFFLE, + 2 => BLOSC_BITSHUFFLE, + _ => BLOSC_NOSHUFFLE, } } -fn internal_decompress(data: &[u8], cname: u8, _nbytes: usize) -> Result, ()> { +fn compcode_from_cname(cname: u8) -> Option { match cname { - BLOSC_BLOSCLZ | BLOSC_LZ4 | BLOSC_LZ4HC => { - lz4_flex::decompress_size_prepended(data).map_err(|_| ()) - } - BLOSC_SNAPPY => { - let mut decoder = snap::raw::Decoder::new(); - decoder.decompress_vec(data).map_err(|_| ()) - } - BLOSC_ZSTD => zstd::decode_all(data).map_err(|_| ()), - BLOSC_ZLIB => { - lz4_flex::decompress_size_prepended(data).map_err(|_| ()) - } - _ => Err(()), + 0 => Some(BLOSC_BLOSCLZ), + 1 => Some(BLOSC_LZ4), + 2 => Some(BLOSC_LZ4HC), + 4 => Some(BLOSC_ZLIB), + 5 => Some(BLOSC_ZSTD), + // 3 was historical snappy; not a standard C-Blosc2 compressor id path here + _ => None, } } @@ -190,134 +50,107 @@ pub fn blosc2_compress<'a>( typesize: usize, ) -> Term<'a> { let clevel = clevel.clamp(0, 9) as u8; - let typesize = typesize.max(1); - let cname = cname.clamp(0, 5) as u8; - let shuffle = shuffle.clamp(0, 2) as u8; - - if data.len() > u32::MAX as usize { - return (atoms::error(), atoms::invalid_data()).encode(env); - } + let typesize = typesize.clamp(1, 255) as i32; + let Some(compcode) = compcode_from_cname(cname as u8) else { + return err(env, atoms::invalid_options()); + }; + let filter = filter_from_shuffle(shuffle.clamp(0, 2) as u8); - if data.is_empty() { - let mut header = vec![0u8; BLOSC_MIN_HEADER_LENGTH]; - header[0] = BLOSC2_MAGIC; - header[1] = BLOSC2_VERSION; - header[2] = 0x01; - header[3] = 0; - header[4..8].copy_from_slice(&0u32.to_le_bytes()); - header[8..12].copy_from_slice(&(BLOSC_MIN_HEADER_LENGTH as u32).to_le_bytes()); - header[12] = cname; - header[13] = clevel; - header[14] = BLOSC_NOSHUFFLE; - header[15] = typesize.max(1) as u8; - return (atoms::ok(), encode_binary(env, &header)).encode(env); + if data.len() > i32::MAX as usize { + return err(env, atoms::invalid_data()); } - let shuffled = match shuffle { - BLOSC_BYTESHUFFLE => byte_shuffle(data.as_slice(), typesize), - BLOSC_BITSHUFFLE => bit_shuffle(data.as_slice(), typesize), - _ => data.as_slice().to_vec(), + let cparams = CParams { + compcode, + clevel, + typesize, + nthreads: 1, + filters: [0, 0, 0, 0, 0, filter], + filters_meta: [0; 6], + ..Default::default() }; - if clevel == 0 { - let payload = data.as_slice(); - let total_len = BLOSC_MIN_HEADER_LENGTH + payload.len(); - let mut output = vec![0u8; total_len]; - output[0] = BLOSC2_MAGIC; - output[1] = BLOSC2_VERSION; - output[2] = 0x01; - output[3] = 0; - output[4..8].copy_from_slice(&(payload.len() as u32).to_le_bytes()); - output[8..12].copy_from_slice(&(total_len as u32).to_le_bytes()); - output[12] = 0; - output[13] = 0; - output[14] = BLOSC_NOSHUFFLE; - output[15] = typesize.max(1) as u8; - output[BLOSC_MIN_HEADER_LENGTH..].copy_from_slice(payload); - return (atoms::ok(), encode_binary(env, &output)).encode(env); - } - - let compressed = match internal_compress(&shuffled, cname, clevel) { - Ok(c) => c, - Err(()) => return (atoms::error(), atoms::compression_failed()).encode(env), + let ctx = match blosc2_create_cctx(cparams) { + Ok(ctx) => ctx, + Err(_) => return err(env, atoms::invalid_options()), }; - if compressed.len() >= shuffled.len() { - let payload = data.as_slice(); - let total_len = BLOSC_MIN_HEADER_LENGTH + payload.len(); - let mut output = vec![0u8; total_len]; - output[0] = BLOSC2_MAGIC; - output[1] = BLOSC2_VERSION; - output[2] = 0x01; - output[3] = 0; - output[4..8].copy_from_slice(&(payload.len() as u32).to_le_bytes()); - output[8..12].copy_from_slice(&(total_len as u32).to_le_bytes()); - output[12] = 0; - output[13] = 0; - output[14] = BLOSC_NOSHUFFLE; - output[15] = typesize.max(1) as u8; - output[BLOSC_MIN_HEADER_LENGTH..].copy_from_slice(payload); - return (atoms::ok(), encode_binary(env, &output)).encode(env); + let src = data.as_slice(); + let srcsize = src.len() as i32; + let mut destsize = (src.len() + BLOSC2_MAX_OVERHEAD) as i32; + if destsize < BLOSC2_MAX_OVERHEAD as i32 { + destsize = BLOSC2_MAX_OVERHEAD as i32; + } + let mut dest = vec![0u8; destsize as usize]; + + let n = blosc2_compress_ctx(&ctx, src, srcsize, &mut dest, destsize); + blosc2_free_ctx(ctx); + + if n < 0 { + return err(env, atoms::compression_failed()); + } + if n == 0 { + // Destination too small — retry with a larger buffer. + let destsize2 = ((src.len() * 2) + BLOSC2_MAX_OVERHEAD).max(BLOSC2_MAX_OVERHEAD) as i32; + let mut dest2 = vec![0u8; destsize2 as usize]; + let cparams2 = CParams { + compcode, + clevel, + typesize, + nthreads: 1, + filters: [0, 0, 0, 0, 0, filter], + ..Default::default() + }; + let Ok(ctx2) = blosc2_create_cctx(cparams2) else { + return err(env, atoms::compression_failed()); + }; + let n2 = blosc2_compress_ctx(&ctx2, src, srcsize, &mut dest2, destsize2); + blosc2_free_ctx(ctx2); + if n2 <= 0 { + return err(env, atoms::compression_failed()); + } + dest2.truncate(n2 as usize); + return ok_binary(env, &dest2); } - let total_len = BLOSC_MIN_HEADER_LENGTH + compressed.len(); - let mut output = vec![0u8; total_len]; - output[0] = BLOSC2_MAGIC; - output[1] = BLOSC2_VERSION; - output[2] = 0x00; - output[3] = 0; - output[4..8].copy_from_slice(&(data.len() as u32).to_le_bytes()); - output[8..12].copy_from_slice(&(total_len as u32).to_le_bytes()); - output[12] = cname; - output[13] = clevel; - output[14] = shuffle; - output[15] = typesize.max(1) as u8; - output[BLOSC_MIN_HEADER_LENGTH..].copy_from_slice(&compressed); - - (atoms::ok(), encode_binary(env, &output)).encode(env) + dest.truncate(n as usize); + ok_binary(env, &dest) } #[rustler::nif(schedule = "DirtyCpu")] -pub fn blosc2_decompress<'a>(env: Env<'a>, data: Binary) -> Term<'a> { - if data.len() < BLOSC_MIN_HEADER_LENGTH { - return (atoms::error(), atoms::invalid_data()).encode(env); - } - - if data[0] != BLOSC2_MAGIC { - return (atoms::error(), atoms::invalid_data()).encode(env); +pub fn blosc2_decompress<'a>(env: Env<'a>, data: Binary, max_output_size: u64) -> Term<'a> { + if data.len() < 16 { + return err(env, atoms::invalid_data()); } + // nbytes at offset 4 (little-endian int32) in the Blosc chunk header. let nbytes = u32::from_le_bytes([data[4], data[5], data[6], data[7]]) as usize; - let cbytes = u32::from_le_bytes([data[8], data[9], data[10], data[11]]) as usize; - let flags = data[2]; - let cname = data[12]; - let _clevel = data[13]; - let shuffle = data[14]; - let typesize = data[15] as usize; - - if cbytes > data.len() { - return (atoms::error(), atoms::invalid_data()).encode(env); + // Cap single-chunk decompress to 1 GiB, and to the caller-supplied limit. + if nbytes > (1usize << 30) || !crate::util::output_within_limit(nbytes, max_output_size) { + return err(env, atoms::output_limit_exceeded()); } - if flags & 0x01 != 0 { - if BLOSC_MIN_HEADER_LENGTH + nbytes > data.len() { - return (atoms::error(), atoms::invalid_data()).encode(env); - } - let result = data[BLOSC_MIN_HEADER_LENGTH..BLOSC_MIN_HEADER_LENGTH + nbytes].to_vec(); - return (atoms::ok(), encode_binary(env, &result)).encode(env); - } - - let payload = &data[BLOSC_MIN_HEADER_LENGTH..cbytes]; - let decompressed = match internal_decompress(payload, cname, nbytes) { - Ok(d) => d, - Err(()) => return (atoms::error(), atoms::decompression_failed()).encode(env), + let dparams = DParams { + nthreads: 1, + ..Default::default() }; - - let result = match shuffle { - BLOSC_BYTESHUFFLE => byte_unshuffle(&decompressed, typesize), - BLOSC_BITSHUFFLE => bit_unshuffle(&decompressed, typesize), - _ => decompressed, + let ctx = match blosc2_create_dctx(dparams) { + Ok(ctx) => ctx, + Err(_) => return err(env, atoms::decompression_failed()), }; - (atoms::ok(), encode_binary(env, &result)).encode(env) -} \ No newline at end of file + let src = data.as_slice(); + let srcsize = src.len() as i32; + let mut dest = vec![0u8; nbytes.max(1)]; + let destsize = dest.len() as i32; + + let n = blosc2_decompress_ctx(&ctx, src, srcsize, &mut dest, destsize); + blosc2_free_ctx(ctx); + + if n < 0 { + return err(env, atoms::decompression_failed()); + } + + dest.truncate(n as usize); + ok_binary(env, &dest) +} diff --git a/native/ex_codecs_native/src/bzip2_codec.rs b/native/ex_codecs_native/src/bzip2_codec.rs index 48d45ee..604bcb9 100644 --- a/native/ex_codecs_native/src/bzip2_codec.rs +++ b/native/ex_codecs_native/src/bzip2_codec.rs @@ -1,11 +1,11 @@ -use rustler::{Binary, Encoder, Env, Term}; +use rustler::{Binary, Env, Term}; use std::io::{Read, Write}; use crate::atoms; -use crate::util::encode_binary; +use crate::util::{err, ok_binary, output_within_limit}; pub fn version() -> String { - "0.4.x".to_string() + "bzip2-0.6/libbz2-rs".to_string() } #[rustler::nif(schedule = "DirtyCpu")] @@ -15,10 +15,8 @@ pub fn bzip2_compress<'a>(env: Env<'a>, data: Binary, block_size: u32) -> Term<' let result: Result, std::io::Error> = (|| { let mut compressed = Vec::with_capacity(data.len() / 2); { - let mut writer = bzip2::write::BzEncoder::new( - &mut compressed, - bzip2::Compression::new(block_size), - ); + let mut writer = + bzip2::write::BzEncoder::new(&mut compressed, bzip2::Compression::new(block_size)); writer.write_all(data.as_slice())?; writer.finish()?; } @@ -26,24 +24,38 @@ pub fn bzip2_compress<'a>(env: Env<'a>, data: Binary, block_size: u32) -> Term<' })(); match result { - Ok(compressed) => (atoms::ok(), encode_binary(env, &compressed)).encode(env), - Err(_) => (atoms::error(), atoms::compression_failed()).encode(env), + Ok(compressed) => ok_binary(env, &compressed), + Err(_) => err(env, atoms::compression_failed()), } } #[rustler::nif(schedule = "DirtyCpu")] -pub fn bzip2_decompress<'a>(env: Env<'a>, data: Binary) -> Term<'a> { - let result: Result, std::io::Error> = (|| { - let mut decompressed = Vec::with_capacity(data.len() * 4); - { - let mut reader = bzip2::read::BzDecoder::new(data.as_slice()); - reader.read_to_end(&mut decompressed)?; +pub fn bzip2_decompress<'a>(env: Env<'a>, data: Binary, max_output_size: u64) -> Term<'a> { + let result: Result, LimitError> = (|| { + let mut decompressed = Vec::with_capacity(data.len().saturating_mul(4).min(64 * 1024)); + let mut reader = bzip2::read::BzDecoder::new(data.as_slice()); + let mut buf = [0u8; 8192]; + loop { + let n = reader.read(&mut buf).map_err(|_| LimitError::Io)?; + if n == 0 { + break; + } + if !output_within_limit(decompressed.len().saturating_add(n), max_output_size) { + return Err(LimitError::Limit); + } + decompressed.extend_from_slice(&buf[..n]); } Ok(decompressed) })(); match result { - Ok(decompressed) => (atoms::ok(), encode_binary(env, &decompressed)).encode(env), - Err(_) => (atoms::error(), atoms::decompression_failed()).encode(env), + Ok(decompressed) => ok_binary(env, &decompressed), + Err(LimitError::Limit) => err(env, atoms::output_limit_exceeded()), + Err(LimitError::Io) => err(env, atoms::decompression_failed()), } -} \ No newline at end of file +} + +enum LimitError { + Limit, + Io, +} diff --git a/native/ex_codecs_native/src/lib.rs b/native/ex_codecs_native/src/lib.rs index b57b8a7..a1e6118 100644 --- a/native/ex_codecs_native/src/lib.rs +++ b/native/ex_codecs_native/src/lib.rs @@ -17,4 +17,4 @@ fn codec_versions() -> std::collections::HashMap<&'static str, String> { versions.insert("bzip2", bzip2_codec::version()); versions.insert("blosc2", blosc2_codec::version()); versions -} \ No newline at end of file +} diff --git a/native/ex_codecs_native/src/lz4_codec.rs b/native/ex_codecs_native/src/lz4_codec.rs index b02f9b0..a46ff5b 100644 --- a/native/ex_codecs_native/src/lz4_codec.rs +++ b/native/ex_codecs_native/src/lz4_codec.rs @@ -1,23 +1,36 @@ -use rustler::{Binary, Encoder, Env, Term}; +use rustler::{Binary, Env, Term}; use crate::atoms; -use crate::util::encode_binary; +use crate::util::{err, ok_binary, output_within_limit}; pub fn version() -> String { - "1.10.x".to_string() + "lz4_flex-0.11".to_string() } #[rustler::nif(schedule = "DirtyCpu")] pub fn lz4_compress<'a>(env: Env<'a>, data: Binary) -> Term<'a> { let compressed = lz4_flex::compress_prepend_size(data.as_slice()); - - (atoms::ok(), encode_binary(env, &compressed)).encode(env) + ok_binary(env, &compressed) } #[rustler::nif(schedule = "DirtyCpu")] -pub fn lz4_decompress<'a>(env: Env<'a>, data: Binary) -> Term<'a> { - match lz4_flex::decompress_size_prepended(data.as_slice()) { - Ok(decompressed) => (atoms::ok(), encode_binary(env, &decompressed)).encode(env), - Err(_) => (atoms::error(), atoms::decompression_failed()).encode(env), +pub fn lz4_decompress<'a>(env: Env<'a>, data: Binary, max_output_size: u64) -> Term<'a> { + let slice = data.as_slice(); + if slice.len() < 4 { + return err(env, atoms::decompression_failed()); + } + let claimed = u32::from_le_bytes([slice[0], slice[1], slice[2], slice[3]]) as usize; + if !output_within_limit(claimed, max_output_size) { + return err(env, atoms::output_limit_exceeded()); } -} \ No newline at end of file + + match lz4_flex::decompress_size_prepended(slice) { + Ok(decompressed) => { + if !output_within_limit(decompressed.len(), max_output_size) { + return err(env, atoms::output_limit_exceeded()); + } + ok_binary(env, &decompressed) + } + Err(_) => err(env, atoms::decompression_failed()), + } +} diff --git a/native/ex_codecs_native/src/snappy_codec.rs b/native/ex_codecs_native/src/snappy_codec.rs index 687d3ab..741bb18 100644 --- a/native/ex_codecs_native/src/snappy_codec.rs +++ b/native/ex_codecs_native/src/snappy_codec.rs @@ -1,26 +1,39 @@ -use rustler::{Binary, Encoder, Env, Term}; +use rustler::{Binary, Env, Term}; use crate::atoms; -use crate::util::encode_binary; +use crate::util::{err, ok_binary, output_within_limit}; pub fn version() -> String { - "1.1.x".to_string() + "snap-1.1".to_string() } #[rustler::nif(schedule = "DirtyCpu")] pub fn snappy_compress<'a>(env: Env<'a>, data: Binary) -> Term<'a> { let mut encoder = snap::raw::Encoder::new(); match encoder.compress_vec(data.as_slice()) { - Ok(compressed) => (atoms::ok(), encode_binary(env, &compressed)).encode(env), - Err(_) => (atoms::error(), atoms::compression_failed()).encode(env), + Ok(compressed) => ok_binary(env, &compressed), + Err(_) => err(env, atoms::compression_failed()), } } #[rustler::nif(schedule = "DirtyCpu")] -pub fn snappy_decompress<'a>(env: Env<'a>, data: Binary) -> Term<'a> { +pub fn snappy_decompress<'a>(env: Env<'a>, data: Binary, max_output_size: u64) -> Term<'a> { + match snap::raw::decompress_len(data.as_slice()) { + Ok(claimed) if !output_within_limit(claimed, max_output_size) => { + return err(env, atoms::output_limit_exceeded()); + } + Ok(_) => {} + Err(_) => return err(env, atoms::decompression_failed()), + } + let mut decoder = snap::raw::Decoder::new(); match decoder.decompress_vec(data.as_slice()) { - Ok(decompressed) => (atoms::ok(), encode_binary(env, &decompressed)).encode(env), - Err(_) => (atoms::error(), atoms::decompression_failed()).encode(env), + Ok(decompressed) => { + if !output_within_limit(decompressed.len(), max_output_size) { + return err(env, atoms::output_limit_exceeded()); + } + ok_binary(env, &decompressed) + } + Err(_) => err(env, atoms::decompression_failed()), } -} \ No newline at end of file +} diff --git a/native/ex_codecs_native/src/util.rs b/native/ex_codecs_native/src/util.rs index fd12713..1c6b25d 100644 --- a/native/ex_codecs_native/src/util.rs +++ b/native/ex_codecs_native/src/util.rs @@ -2,12 +2,33 @@ use rustler::{Binary, Encoder, Env, OwnedBinary, Term}; use crate::atoms; -pub fn encode_binary<'a>(env: Env<'a>, data: &[u8]) -> Term<'a> { +/// Returns true when `len` fits within the caller-supplied decompress ceiling. +pub fn output_within_limit(len: usize, max_output_size: u64) -> bool { + (len as u64) <= max_output_size +} + +/// Copy bytes into an Erlang binary term. +/// +/// Returns `Ok(binary_term)` or `Err` when allocation fails so callers never +/// nest `{:ok, {:error, _}}`. +pub fn encode_binary<'a>(env: Env<'a>, data: &[u8]) -> Result, ()> { match OwnedBinary::new(data.len()) { Some(mut owned) => { owned.as_mut_slice().copy_from_slice(data); - Binary::from_owned(owned, env).encode(env) + Ok(Binary::from_owned(owned, env).encode(env)) } - None => (atoms::error(), atoms::invalid_data()).encode(env), + None => Err(()), + } +} + +/// Encode `{:ok, binary}` or `{:error, reason}` for NIF returns. +pub fn ok_binary<'a>(env: Env<'a>, data: &[u8]) -> Term<'a> { + match encode_binary(env, data) { + Ok(bin) => (atoms::ok(), bin).encode(env), + Err(()) => (atoms::error(), atoms::invalid_data()).encode(env), } -} \ No newline at end of file +} + +pub fn err<'a>(env: Env<'a>, reason: rustler::types::atom::Atom) -> Term<'a> { + (atoms::error(), reason).encode(env) +} diff --git a/native/ex_codecs_native/src/zstd_codec.rs b/native/ex_codecs_native/src/zstd_codec.rs index 480cb74..3142ca6 100644 --- a/native/ex_codecs_native/src/zstd_codec.rs +++ b/native/ex_codecs_native/src/zstd_codec.rs @@ -1,26 +1,46 @@ -use rustler::{Binary, Encoder, Env, Term}; +use rustler::{Binary, Env, Term}; +use std::io::Read; use crate::atoms; -use crate::util::encode_binary; +use crate::util::{err, ok_binary, output_within_limit}; + +use structured_zstd::encoding::{compress_to_vec, CompressionLevel}; pub fn version() -> String { - "1.5.x".to_string() + "structured-zstd-0.0.48".to_string() +} + +fn compression_level(level: i32) -> CompressionLevel { + // Pure-Rust encoder with numeric levels 1..=22 (no C libzstd). + CompressionLevel::from_level(level.clamp(1, 22)) } #[rustler::nif(schedule = "DirtyCpu")] pub fn zstd_compress<'a>(env: Env<'a>, data: Binary, level: i32) -> Term<'a> { - let level = level.clamp(1, 22); - - match zstd::bulk::compress(data.as_slice(), level) { - Ok(compressed) => (atoms::ok(), encode_binary(env, &compressed)).encode(env), - Err(_) => (atoms::error(), atoms::compression_failed()).encode(env), - } + let compressed = compress_to_vec(data.as_slice(), compression_level(level)); + ok_binary(env, &compressed) } #[rustler::nif(schedule = "DirtyCpu")] -pub fn zstd_decompress<'a>(env: Env<'a>, data: Binary) -> Term<'a> { - match zstd::decode_all(data.as_slice()) { - Ok(decompressed) => (atoms::ok(), encode_binary(env, &decompressed)).encode(env), - Err(_) => (atoms::error(), atoms::decompression_failed()).encode(env), +pub fn zstd_decompress<'a>(env: Env<'a>, data: Binary, max_output_size: u64) -> Term<'a> { + match structured_zstd::decoding::StreamingDecoder::new(data.as_slice()) { + Ok(mut decoder) => { + let mut out = Vec::new(); + let mut buf = [0u8; 8192]; + loop { + match decoder.read(&mut buf) { + Ok(0) => break, + Ok(n) => { + if !output_within_limit(out.len().saturating_add(n), max_output_size) { + return err(env, atoms::output_limit_exceeded()); + } + out.extend_from_slice(&buf[..n]); + } + Err(_) => return err(env, atoms::decompression_failed()), + } + } + ok_binary(env, &out) + } + Err(_) => err(env, atoms::decompression_failed()), } -} \ No newline at end of file +} diff --git a/priv/examples/spatial/cube_corners.ply b/priv/examples/spatial/cube_corners.ply new file mode 100644 index 0000000..45f3d29 --- /dev/null +++ b/priv/examples/spatial/cube_corners.ply @@ -0,0 +1,15 @@ +ply +format ascii 1.0 +comment ex_codecs example point cloud +element vertex 4 +property float x +property float y +property float z +property uchar red +property uchar green +property uchar blue +end_header +0.0 0.0 0.0 255 0 0 +1.0 0.0 0.0 0 255 0 +0.0 1.0 0.0 0 0 255 +0.0 0.0 1.0 255 255 0 diff --git a/priv/examples/spatial/two_gaussians.ply b/priv/examples/spatial/two_gaussians.ply new file mode 100644 index 0000000..b09a514 --- /dev/null +++ b/priv/examples/spatial/two_gaussians.ply @@ -0,0 +1,21 @@ +ply +format ascii 1.0 +comment ex_codecs example gaussian splat +element vertex 2 +property float x +property float y +property float z +property float f_dc_0 +property float f_dc_1 +property float f_dc_2 +property float opacity +property float scale_0 +property float scale_1 +property float scale_2 +property float rot_0 +property float rot_1 +property float rot_2 +property float rot_3 +end_header +0.0 0.0 0.0 0.5 0.4 0.3 0.9 0.1 0.1 0.1 1.0 0.0 0.0 0.0 +1.0 1.0 1.0 0.2 0.6 0.8 0.7 0.2 0.2 0.2 0.9 0.1 0.0 0.0 diff --git a/test/ex_codecs/codec_registry_test.exs b/test/ex_codecs/codec_registry_test.exs index c210769..836235b 100644 --- a/test/ex_codecs/codec_registry_test.exs +++ b/test/ex_codecs/codec_registry_test.exs @@ -4,104 +4,117 @@ defmodule ExCodecs.CodecRegistryTest do alias ExCodecs.CodecRegistry setup do - CodecRegistry.start_link([]) - :ok + # Use the application-owned registry; clean up any names we register. + names = + for i <- 1..20 do + :"registry_test_#{System.unique_integer([:positive])}_#{i}" + end + + on_exit(fn -> + Enum.each(names, &CodecRegistry.unregister/1) + end) + + {:ok, names: names} end describe "register/3" do - test "registers a valid codec module" do - assert :ok = CodecRegistry.register(:zstd, ExCodecs.Compression.Zstd, :compression) - assert {:ok, {ExCodecs.Compression.Zstd, :compression, _}} = CodecRegistry.lookup(:zstd) + test "registers a valid codec module", %{names: [name | _]} do + assert :ok = CodecRegistry.register(name, ExCodecs.Compression.Zstd, :compression) + assert {:ok, {ExCodecs.Compression.Zstd, :compression, _}} = CodecRegistry.lookup(name) end - test "returns error for invalid module" do + test "returns error for invalid module", %{names: [name | _]} do assert {:error, {:invalid_codec_module, NonexistentModule}} = - CodecRegistry.register(:fake, NonexistentModule, :compression) + CodecRegistry.register(name, NonexistentModule, :compression) end - test "returns error for module missing encode/2" do + test "returns error for module missing encode/2", %{names: [name | _]} do defmodule NoEncode do def decode(_data, _opts), do: {:ok, ""} end assert {:error, {:invalid_codec_module, NoEncode}} = - CodecRegistry.register(:no_encode, NoEncode, :compression) + CodecRegistry.register(name, NoEncode, :compression) end - test "returns error for module missing decode/2" do + test "returns error for module missing decode/2", %{names: [name | _]} do defmodule NoDecode do def encode(_data, _opts), do: {:ok, ""} end assert {:error, {:invalid_codec_module, NoDecode}} = - CodecRegistry.register(:no_decode, NoDecode, :compression) + CodecRegistry.register(name, NoDecode, :compression) end end describe "register_unavailable/2" do - test "registers codec as unavailable" do - assert :ok = CodecRegistry.register_unavailable(:future_codec, :compression) - assert {:ok, {nil, :compression, info}} = CodecRegistry.lookup(:future_codec) + test "registers codec as unavailable", %{names: [name | _]} do + assert :ok = CodecRegistry.register_unavailable(name, :compression) + assert {:ok, {nil, :compression, info}} = CodecRegistry.lookup(name) assert info.module == nil assert info.native? == false end - test "unavailable codec is not in available_codecs" do - CodecRegistry.register_unavailable(:unavail, :compression) - refute :unavail in CodecRegistry.available_codecs() + test "unavailable codec is not in available_codecs", %{names: [name | _]} do + CodecRegistry.register_unavailable(name, :compression) + refute name in CodecRegistry.available_codecs() end - test "unavailable codec shows in all_codecs" do - CodecRegistry.register_unavailable(:unavail, :compression) - assert :unavail in CodecRegistry.all_codecs() + test "unavailable codec shows in all_codecs", %{names: [name | _]} do + CodecRegistry.register_unavailable(name, :compression) + assert name in CodecRegistry.all_codecs() end end describe "lookup/1" do test "returns error for unknown codec" do - assert {:error, :unsupported_codec} = CodecRegistry.lookup(:nonexistent) + assert {:error, :unsupported_codec} = CodecRegistry.lookup(:nonexistent_codec_xyz) end end describe "available_codecs/0" do - test "returns list of available codec names" do - CodecRegistry.register(:zstd, ExCodecs.Compression.Zstd, :compression) - CodecRegistry.register(:lz4, ExCodecs.Compression.Lz4, :compression) + test "includes built-in codecs from application start" do codecs = CodecRegistry.available_codecs() assert :zstd in codecs assert :lz4 in codecs + assert :ply in codecs + assert :spatial_binary in codecs + assert :gsplat in codecs + end + + test "filters available codecs by category" do + assert CodecRegistry.available_codecs(:spatial) == [:gsplat, :ply, :spatial_binary] + assert :zstd in CodecRegistry.available_codecs(:compression) + refute :ply in CodecRegistry.available_codecs(:compression) end end describe "all_codecs/0" do - test "returns all codecs including unavailable" do - CodecRegistry.register(:zstd, ExCodecs.Compression.Zstd, :compression) - CodecRegistry.register_unavailable(:future_codec, :compression) + test "returns all codecs including unavailable", %{names: [name | _]} do + CodecRegistry.register_unavailable(name, :compression) all = CodecRegistry.all_codecs() assert :zstd in all - assert :future_codec in all + assert name in all end end describe "supports?/1" do test "returns true for available codecs" do - CodecRegistry.register(:zstd, ExCodecs.Compression.Zstd, :compression) assert CodecRegistry.supports?(:zstd) == true end - test "returns false for unavailable codecs" do - CodecRegistry.register_unavailable(:future_codec, :compression) - assert CodecRegistry.supports?(:future_codec) == false + test "returns false for unavailable codecs", %{names: [name | _]} do + CodecRegistry.register_unavailable(name, :compression) + assert CodecRegistry.supports?(name) == false end test "returns false for unknown codecs" do - assert CodecRegistry.supports?(:nonexistent) == false + assert CodecRegistry.supports?(:nonexistent_codec_xyz) == false end end describe "codec_info/1" do test "returns info for registered codec" do - CodecRegistry.register(:zstd, ExCodecs.Compression.Zstd, :compression) assert {:ok, info} = CodecRegistry.codec_info(:zstd) assert info.name == :zstd assert info.category == :compression @@ -109,14 +122,20 @@ defmodule ExCodecs.CodecRegistryTest do end test "returns error for unknown codec" do - assert {:error, :unsupported_codec} = CodecRegistry.codec_info(:nonexistent) + assert {:error, :unsupported_codec} = CodecRegistry.codec_info(:nonexistent_codec_xyz) + end + + test "returns interface metadata for spatial formats" do + assert {:ok, info} = CodecRegistry.codec_info(:ply) + assert info.category == :spatial + assert info.interface == :spatial + assert info.configurable? + assert info.version == "PLY 1.0" end end describe "codecs_by_category/1" do test "returns codecs in a category" do - CodecRegistry.register(:zstd, ExCodecs.Compression.Zstd, :compression) - CodecRegistry.register(:lz4, ExCodecs.Compression.Lz4, :compression) codecs = CodecRegistry.codecs_by_category(:compression) assert length(codecs) >= 2 names = Enum.map(codecs, & &1.name) @@ -125,7 +144,21 @@ defmodule ExCodecs.CodecRegistryTest do end test "returns empty list for unknown category" do - assert [] = CodecRegistry.codecs_by_category(:nonexistent) + assert [] = CodecRegistry.codecs_by_category(:nonexistent_category_xyz) + end + + test "returns shared-catalog spatial entries" do + codecs = CodecRegistry.codecs_by_category(:spatial) + assert Enum.map(codecs, & &1.name) == [:gsplat, :ply, :spatial_binary] + assert Enum.all?(codecs, &(&1.interface == :spatial)) + end + end + + describe "unregister/1" do + test "removes a codec entry", %{names: [name | _]} do + assert :ok = CodecRegistry.register_unavailable(name, :compression) + assert :ok = CodecRegistry.unregister(name) + assert {:error, :unsupported_codec} = CodecRegistry.lookup(name) end end end diff --git a/test/ex_codecs/codec_test.exs b/test/ex_codecs/codec_test.exs index 0e3c961..7f888ac 100644 --- a/test/ex_codecs/codec_test.exs +++ b/test/ex_codecs/codec_test.exs @@ -17,6 +17,7 @@ defmodule ExCodecs.CodecTest do assert codec.name == :zstd assert codec.category == :compression + assert codec.interface == :binary assert codec.module == ExCodecs.Compression.Zstd assert codec.native? == true assert codec.streaming? == true @@ -27,6 +28,7 @@ defmodule ExCodecs.CodecTest do test "has default values" do codec = %Codec{name: :test} assert codec.category == nil + assert codec.interface == :binary assert codec.module == nil assert codec.native? == nil assert codec.streaming? == nil diff --git a/test/ex_codecs/compression/blosc2_interop_test.exs b/test/ex_codecs/compression/blosc2_interop_test.exs new file mode 100644 index 0000000..a0b373d --- /dev/null +++ b/test/ex_codecs/compression/blosc2_interop_test.exs @@ -0,0 +1,62 @@ +defmodule ExCodecs.Compression.Blosc2InteropTest do + use ExUnit.Case, async: true + + @moduletag :blosc2_interop + + @fixture_dir Path.expand("../../fixtures/blosc2", __DIR__) + + @goldens [ + "lz4_noshuffle_t1.bin", + "lz4_shuffle_t8.bin", + "zstd_shuffle_t8.bin", + "blosclz_noshuffle.bin", + "lz4hc_noshuffle.bin", + "zlib_noshuffle.bin", + "lz4_bitshuffle_t8.bin" + ] + + describe "python-blosc2 golden chunks" do + for name <- @goldens do + test "decompresses #{name}" do + bin = File.read!(Path.join(@fixture_dir, unquote(name))) + src = File.read!(Path.join(@fixture_dir, String.replace(unquote(name), ".bin", ".src"))) + + assert {:ok, decompressed} = ExCodecs.decode(:blosc2, bin) + assert decompressed == src + end + end + end + + describe "round-trip all official cnames" do + test "blosclz lz4 lz4hc zlib zstd with filters" do + data = :crypto.strong_rand_bytes(4096) + floats = for i <- 1..512, into: <<>>, do: <> + + for cname <- [:blosclz, :lz4, :lz4hc, :zlib, :zstd] do + assert {:ok, c} = ExCodecs.encode(:blosc2, data, cname: cname, shuffle: :none, typesize: 1) + assert {:ok, ^data} = ExCodecs.decode(:blosc2, c) + end + + for {shuffle, typesize, payload} <- [ + {:byte, 8, floats}, + {:bit, 8, floats}, + {:none, 1, data} + ] do + assert {:ok, c} = + ExCodecs.encode(:blosc2, payload, + cname: :lz4, + clevel: 5, + shuffle: shuffle, + typesize: typesize + ) + + assert {:ok, ^payload} = ExCodecs.decode(:blosc2, c) + end + end + + test "rejects snappy" do + assert {:error, %ExCodecs.Error{reason: :invalid_options}} = + ExCodecs.encode(:blosc2, "x", cname: :snappy) + end + end +end diff --git a/test/ex_codecs/compression/blosc2_test.exs b/test/ex_codecs/compression/blosc2_test.exs index 740d860..49790d1 100644 --- a/test/ex_codecs/compression/blosc2_test.exs +++ b/test/ex_codecs/compression/blosc2_test.exs @@ -149,9 +149,35 @@ defmodule ExCodecs.Compression.Blosc2Test do assert info.name == :blosc2 assert info.category == :compression assert info.configurable? == true - assert info.streaming? == true + assert info.streaming? == false assert info.native? == true assert is_binary(info.version) end end + + describe "c-blosc2 cnames" do + test "accepts blosclz and lz4hc" do + data = :crypto.strong_rand_bytes(1024) + + assert {:ok, c} = Blosc2.encode(data, cname: :blosclz, shuffle: :none) + assert {:ok, ^data} = Blosc2.decode(c, []) + + assert {:ok, c} = Blosc2.encode(data, cname: :lz4hc, shuffle: :none) + assert {:ok, ^data} = Blosc2.decode(c, []) + end + + test "round-trips official cnames" do + data = :crypto.strong_rand_bytes(2048) + + for cname <- [:blosclz, :lz4, :lz4hc, :zlib, :zstd] do + assert {:ok, compressed} = Blosc2.encode(data, cname: cname, shuffle: :none, typesize: 1) + assert {:ok, ^data} = Blosc2.decode(compressed, []) + end + end + + test "rejects snappy" do + assert {:error, %ExCodecs.Error{reason: :invalid_options}} = + Blosc2.encode("data", cname: :snappy) + end + end end diff --git a/test/ex_codecs/compression/lz4_test.exs b/test/ex_codecs/compression/lz4_test.exs index 4bdd1c7..228b048 100644 --- a/test/ex_codecs/compression/lz4_test.exs +++ b/test/ex_codecs/compression/lz4_test.exs @@ -48,6 +48,14 @@ defmodule ExCodecs.Compression.Lz4Test do test "returns error for corrupt data" do assert {:error, _} = Lz4.decode(<<0, 1, 2, 3>>, []) end + + test "rejects decompress when max_output_size is too small" do + data = String.duplicate("ab", 10_000) + {:ok, compressed} = Lz4.encode(data, []) + + assert {:error, %ExCodecs.Error{reason: :output_limit_exceeded}} = + Lz4.decode(compressed, max_output_size: 16) + end end describe "__codec_info__/0" do diff --git a/test/ex_codecs/compression/version_and_error_test.exs b/test/ex_codecs/compression/version_and_error_test.exs index 77bc1a9..32f67a5 100644 --- a/test/ex_codecs/compression/version_and_error_test.exs +++ b/test/ex_codecs/compression/version_and_error_test.exs @@ -11,7 +11,7 @@ defmodule ExCodecs.Compression.CodecVersionTest do assert info.category == :compression assert info.module == Zstd assert info.native? == true - assert info.streaming? == true + assert info.streaming? == false assert info.configurable? == true assert is_binary(info.version) end @@ -52,7 +52,7 @@ defmodule ExCodecs.Compression.CodecVersionTest do assert info.category == :compression assert info.module == Blosc2 assert info.native? == true - assert info.streaming? == true + assert info.streaming? == false assert info.configurable? == true assert is_binary(info.version) end @@ -80,8 +80,10 @@ defmodule ExCodecs.Compression.CodecVersionTest do end test "blosc2 returns error for invalid magic" do - assert {:error, %Error{reason: :invalid_data}} = + assert {:error, %Error{reason: reason}} = ExCodecs.decode(:blosc2, <<0xFF, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>) + + assert reason in [:invalid_data, :decompression_failed] end test "blosc2 validates cname option" do diff --git a/test/ex_codecs/compression/zstd_test.exs b/test/ex_codecs/compression/zstd_test.exs index 5ef78c3..50c1b7f 100644 --- a/test/ex_codecs/compression/zstd_test.exs +++ b/test/ex_codecs/compression/zstd_test.exs @@ -20,12 +20,32 @@ defmodule ExCodecs.Compression.ZstdTest do end test "higher levels produce smaller or equal output" do - data = :crypto.strong_rand_bytes(10_000) + # Random data is poorly compressible; use repetitive payload so buckets matter. + data = String.duplicate("AAAA", 5_000) {:ok, c1} = Zstd.encode(data, level: 1) {:ok, c19} = Zstd.encode(data, level: 19) assert byte_size(c19) <= byte_size(c1) end + test "level buckets produce distinct frames for compressible data" do + # Varied compressible payload so numeric levels diverge. + data = + for i <- 1..8_000, into: <<>> do + <> + end + + {:ok, l1} = Zstd.encode(data, level: 1) + {:ok, l3} = Zstd.encode(data, level: 3) + {:ok, l15} = Zstd.encode(data, level: 15) + + distinct = MapSet.new([l1, l3, l15]) + assert MapSet.size(distinct) >= 2 + + for frame <- [l1, l3, l15] do + assert {:ok, ^data} = Zstd.decode(frame, []) + end + end + test "level effectiveness: higher levels produce smaller or equal output for compressible data" do data = String.duplicate("Hello, World! ", 10_000) {:ok, c1} = Zstd.encode(data, level: 1) @@ -83,6 +103,21 @@ defmodule ExCodecs.Compression.ZstdTest do test "returns error for invalid data" do assert {:error, _} = Zstd.decode("not compressed", []) end + + test "rejects decompress when max_output_size is too small" do + data = String.duplicate("AAAA", 256) + {:ok, compressed} = Zstd.encode(data, []) + + assert {:error, %ExCodecs.Error{reason: :output_limit_exceeded}} = + Zstd.decode(compressed, max_output_size: 8) + end + + test "rejects invalid max_output_size" do + {:ok, compressed} = Zstd.encode("hi", []) + + assert {:error, %ExCodecs.Error{reason: :invalid_options}} = + Zstd.decode(compressed, max_output_size: 0) + end end describe "__codec_info__/0" do @@ -91,7 +126,7 @@ defmodule ExCodecs.Compression.ZstdTest do assert info.name == :zstd assert info.category == :compression assert info.native? == true - assert info.streaming? == true + assert info.streaming? == false assert info.configurable? == true end end diff --git a/test/ex_codecs/coverage_boost_test.exs b/test/ex_codecs/coverage_boost_test.exs new file mode 100644 index 0000000..ceac9c4 --- /dev/null +++ b/test/ex_codecs/coverage_boost_test.exs @@ -0,0 +1,435 @@ +defmodule ExCodecs.CoverageBoostTest do + use ExUnit.Case, async: true + + alias ExCodecs.Compression.{Bzip2, Lz4, Snappy, Zstd} + alias ExCodecs.NIF + alias ExCodecs.Spatial + alias ExCodecs.Spatial.Codec.{Binary, Gsplat, PLY} + alias ExCodecs.Spatial.{Gaussian, GaussianCloud, Point, PointCloud} + alias ExCodecs.Spatial.Stream, as: SpatialStream + + describe "point cloud bounds expansion and add_points" do + test "expands existing bounds and bulk-adds points" do + cloud = PointCloud.new([Point.new(0, 0, 0)]) + cloud = PointCloud.add_point(cloud, Point.new(2, -1, 3)) + assert cloud.bounds.min_x == 0.0 + assert cloud.bounds.max_x == 2.0 + assert cloud.bounds.min_y == -1.0 + assert cloud.bounds.max_z == 3.0 + + cloud = PointCloud.add_points(cloud, [Point.new(5, 5, 5), Point.new(-2, 0, 0)]) + assert PointCloud.size(cloud) == 4 + assert cloud.bounds.max_x == 5.0 + assert cloud.bounds.min_x == -2.0 + end + end + + describe "compression invalid_data catch-alls" do + test "encode/decode reject non-binaries" do + for {mod, codec} <- [ + {Zstd, :zstd}, + {Lz4, :lz4}, + {Snappy, :snappy}, + {Bzip2, :bzip2} + ] do + assert {:error, %ExCodecs.Error{reason: :invalid_data, codec: ^codec}} = + mod.encode(123, []) + + assert {:error, %ExCodecs.Error{reason: :invalid_data, codec: ^codec}} = + mod.decode(123, []) + end + end + end + + describe "top-level catch-alls and NIF helpers" do + test "encode/decode reject completely invalid argument shapes" do + assert {:error, %ExCodecs.Error{reason: :invalid_data}} = + ExCodecs.encode("not-an-atom", "data") + + assert {:error, %ExCodecs.Error{reason: :invalid_data}} = + ExCodecs.decode("not-an-atom", "data") + end + + test "NIF wrap and safe_call edge paths" do + assert NIF.default_max_output_size() == 268_435_456 + + assert {:error, %ExCodecs.Error{reason: :invalid_data, message: message}} = + NIF.wrap(:zstd, :unexpected) + + assert message =~ "Unexpected NIF return" + + assert {:error, %ExCodecs.Error{reason: :nif_not_loaded, codec: :lz4}} = + NIF.safe_call(:lz4, fn -> raise ErlangError, original: :nif_not_loaded end) + + assert {:error, %ExCodecs.Error{reason: :invalid_data, codec: :lz4, details: :boom}} = + NIF.safe_call(:lz4, fn -> raise ErlangError, original: :boom end) + + assert {:error, %ExCodecs.Error{reason: :invalid_data, codec: :snappy, message: msg}} = + NIF.safe_call(:snappy, fn -> raise ArgumentError, message: "bad arg" end) + + assert msg =~ "bad arg" + end + end + + describe "spatial stream edges" do + test "missing format and file/binary source resolution" do + assert [{:error, %{reason: :invalid_options}}] = + SpatialStream.decode(<<"ply">>, []) |> Enum.to_list() + + assert {:error, %{reason: :invalid_options}} = + SpatialStream.encode([Point.new(0, 0, 0)], []) + + path = + Path.join(System.tmp_dir!(), "ex_codecs_cov_#{System.unique_integer([:positive])}.excp") + + on_exit(fn -> File.rm(path) end) + + cloud = PointCloud.new([Point.new(1, 2, 3)]) + assert :ok = SpatialStream.encode_to_file(cloud, path, format: :spatial_binary) + + assert [%Point{}] = + Spatial.stream_decode(path, format: :spatial_binary, source: :file) + |> Enum.to_list() + + {:ok, bin} = Spatial.encode(cloud, format: :spatial_binary) + + assert [%Point{}] = + Spatial.stream_decode(bin, format: :spatial_binary, source: :binary) + |> Enum.to_list() + + assert [{:error, %{reason: :io_error}}] = + Spatial.stream_decode("/no/such/ex_codecs_file.excp", + format: :spatial_binary, + source: :file + ) + |> Enum.to_list() + + assert [{:error, %{reason: :io_error}}] = + Spatial.stream_decode("/no/such/ex_codecs_file.gspl", + format: :gsplat, + source: :file + ) + |> Enum.to_list() + end + end + + describe "binary codec nil-color and truncation" do + test "encodes mixed color presence and rejects truncated payloads" do + rgb_cloud = + PointCloud.new([ + Point.new(0, 0, 0, color: {1, 2, 3}), + Point.new(1, 1, 1) + ]) + + assert {:ok, rgb_bin} = Binary.encode(rgb_cloud) + assert {:ok, _} = Binary.decode(rgb_bin) + + rgba_cloud = + PointCloud.new([ + Point.new(0, 0, 0, color: {1, 2, 3, 4}), + Point.new(1, 1, 1, color: {5, 6, 7}), + Point.new(2, 2, 2) + ]) + + assert {:ok, rgba_bin} = Binary.encode(rgba_cloud) + assert {:ok, decoded} = Binary.decode(rgba_bin) + assert length(decoded.points) == 3 + + normal_cloud = + PointCloud.new([ + Point.new(0, 0, 0, normal: {0.0, 1.0, 0.0}), + Point.new(1, 1, 1) + ]) + + assert {:ok, _} = Binary.encode(normal_cloud) + + # flags: color + assert {:error, %{message: message}} = + Binary.decode( + <<"EXCP", 1::little-16, 1::little-16, 1::little-64, 0::float-32-little, + 0::float-32-little, 0::float-32-little, 1, 2>> + ) + + assert message =~ "Truncated RGB" + + # flags: alpha + assert {:error, %{message: message}} = + Binary.decode( + <<"EXCP", 1::little-16, 3::little-16, 1::little-64, 0::float-32-little, + 0::float-32-little, 0::float-32-little, 1, 2, 3>> + ) + + assert message =~ "Truncated RGBA" + + # flags: normal + assert {:error, %{message: message}} = + Binary.decode( + <<"EXCP", 1::little-16, 4::little-16, 1::little-64, 0::float-32-little, + 0::float-32-little, 0::float-32-little, 1, 2, 3>> + ) + + assert message =~ "Truncated normal" + end + end + + describe "gsplat SH list shapes" do + test "encodes flat SH lists and rejects truncated SH payloads" do + g = + struct(Gaussian, + position: {0.0, 0.0, 0.0}, + sh: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6] + ) + + cloud = GaussianCloud.new([g]) + assert {:ok, bin} = Gsplat.encode(cloud) + assert {:ok, decoded} = Gsplat.decode(bin) + assert hd(decoded.gaussians).sh != nil + + # Valid header with one gaussian requiring SH floats that are missing. + header = <<"GSPL", 1::little-16, 0::little-16, 1::little-64, 3::little-16>> + base = :binary.copy(<<0::float-32-little>>, 14) + assert {:error, %{message: message}} = Gsplat.decode(header <> base <> <<1, 2>>) + assert message =~ "Truncated SH" + end + end + + describe "PLY typed encode/decode coverage" do + test "format aliases, property aliases, and mixed binary types" do + cloud = PointCloud.new([Point.new(1, 2, 3)]) + + assert {:ok, _} = PLY.encode(cloud, format: :binary_le) + assert {:ok, _} = PLY.encode(cloud, format: :little) + assert {:ok, _} = PLY.encode(cloud, format: :big) + assert {:ok, _} = PLY.encode(cloud, format: :unknown_defaults_ascii) + + assert {:error, %{reason: :invalid_data}} = PLY.decode(123) + + ascii = """ + ply + format ascii 1.0 + element vertex 1 + property float32 x + property float32 y + property float32 z + property int8 tiny + property uint8 utiny + property int16 s16 + property uint16 u16 + property int32 i32 + property uint32 u32 + property float64 d + end_header + 1 2 3 -1 2 -3 4 -5 6 7.5 + """ + + assert {:ok, decoded} = PLY.decode(ascii) + attrs = hd(decoded.points).attributes + assert attrs["tiny"] == -1 + assert attrs["utiny"] == 2 + assert attrs["d"] == 7.5 + + le_header = """ + ply + format binary_little_endian 1.0 + element vertex 1 + property double x + property double y + property double z + property char tiny + property ushort code + property short delta + property uint big + property int label + end_header + """ + + le_body = + <<1.0::little-float-64, 2.0::little-float-64, 3.0::little-float-64, -2::signed-8, + 9::little-unsigned-16, -4::little-signed-16, 11::little-unsigned-32, + 12::little-signed-32>> + + assert {:ok, le_cloud} = PLY.decode(le_header <> le_body) + assert hd(le_cloud.points).attributes["label"] == 12 + + be_header = """ + ply + format binary_big_endian 1.0 + element vertex 1 + property double x + property double y + property double z + property short delta + property uint big + property int label + end_header + """ + + be_body = + <<1.0::big-float-64, 2.0::big-float-64, 3.0::big-float-64, -4::big-signed-16, + 11::big-unsigned-32, 12::big-signed-32>> + + assert {:ok, be_cloud} = PLY.decode(be_header <> be_body) + assert hd(be_cloud.points).attributes["big"] == 11 + end + + test "encode rescue paths, header helpers, and stream sources" do + bad_point_cloud = + PointCloud.new([Point.new(0, 0, 0, attributes: %{"intensity" => :not_a_number})]) + + assert {:error, %{message: message}} = PLY.encode(bad_point_cloud, format: :binary) + assert message =~ "PLY encode failed" + + bad_gaussian = + %GaussianCloud{ + gaussians: [ + struct(Gaussian, position: {0.0, 0.0, 0.0}, sh: :not_a_list) + ] + } + + assert {:error, %{message: message}} = PLY.encode(bad_gaussian) + assert message =~ "PLY encode failed" + + assert {:ok, _parsed, _body} = + PLY.decode_header_and_body(""" + ply + format ascii 1.0 + element vertex 0 + end_header + """) + + path = + Path.join(System.tmp_dir!(), "ex_codecs_ply_cov_#{System.unique_integer([:positive])}.ply") + + on_exit(fn -> File.rm(path) end) + {:ok, bin} = PLY.encode(PointCloud.new([Point.new(0, 0, 1)])) + File.write!(path, bin) + + assert [%Point{}] = PLY.stream_decode(path, source: :file) |> Enum.to_list() + assert [%Point{}] = PLY.stream_decode(bin, source: :binary) |> Enum.to_list() + + assert [{:error, %{reason: :io_error}}] = + PLY.stream_decode("/no/such/ply_cov.ply", source: :file) |> Enum.to_list() + + # Corrupt file contents through the file stream path. + File.write!(path, "not-ply") + + assert [{:error, %{reason: :invalid_data}}] = + PLY.stream_decode(path, source: :file) |> Enum.to_list() + end + + test "gaussian stream decode and sparse attributes" do + gcloud = GaussianCloud.new([Gaussian.new({0.0, 1.0, 2.0})]) + {:ok, gbin} = PLY.encode(gcloud, format: :ascii) + assert [%Gaussian{}] = PLY.stream_decode(gbin) |> Enum.to_list() + + cloud = + GaussianCloud.new([ + struct(Gaussian, position: {0.0, 1.0, 2.0}, sh: nil), + struct(Gaussian, position: {1.0, 1.0, 1.0}, sh: []) + ]) + + assert {:ok, bin} = PLY.encode(cloud, format: :ascii) + assert {:ok, _} = PLY.decode(bin, as: :gaussian_cloud) + + assert {:ok, gspl} = + Gsplat.encode( + GaussianCloud.new([struct(Gaussian, position: {0.0, 0.0, 0.0}, sh: [])]) + ) + + assert {:ok, _} = Gsplat.decode(gspl) + + sparse = + PointCloud.new([ + Point.new(0, 0, 0, attributes: %{"intensity" => 1.0}), + Point.new(1, 1, 1) + ]) + + assert {:ok, _} = PLY.encode(sparse, format: :ascii) + + assert {:ok, weird_cloud} = + PLY.decode(""" + ply + format ascii 1.0 + element vertex 1 + property float x + property float y + property float z + property float weird + end_header + 1 2 3 not-a-number + """) + + assert hd(weird_cloud.points).attributes["weird"] == 0.0 + + assert {:ok, _} = + PLY.decode("ply\nformat ascii 1.0\nelement vertex 0\nend_header") + + assert {:error, _} = + PLY.decode(""" + ply + format ascii 1.0 + element vertex 1 + property float x + property float y + property float z + property notatype name + end_header + 1 2 3 4 + """) + + assert {:error, _} = + PLY.decode(""" + ply + format ascii 1.0 + element vertex nope + property float x + property float y + property float z + end_header + """) + + assert {:error, _} = + PLY.decode(""" + ply + format ascii 1.0 + element vertex -1 + property float x + property float y + property float z + end_header + """) + + # Extra tokens on the element vertex line. + assert {:error, %{message: "Malformed element vertex line in PLY"}} = + PLY.decode(""" + ply + format ascii 1.0 + element vertex 1 extra + property float x + end_header + 1 + """) + + # Property line missing its name. + assert {:error, %{message: "Invalid PLY property"}} = + PLY.decode(""" + ply + format ascii 1.0 + element vertex 1 + property float + end_header + 1 + """) + + path = + Path.join(System.tmp_dir!(), "ex_codecs_auto_#{System.unique_integer([:positive])}.excp") + + on_exit(fn -> File.rm(path) end) + {:ok, excp} = Spatial.encode(PointCloud.new([Point.new(1, 2, 3)]), format: :spatial_binary) + File.write!(path, excp) + + assert [%Point{}] = + Spatial.stream_decode(path, format: :spatial_binary) |> Enum.to_list() + end + end +end diff --git a/test/ex_codecs/documentation_test.exs b/test/ex_codecs/documentation_test.exs new file mode 100644 index 0000000..c631844 --- /dev/null +++ b/test/ex_codecs/documentation_test.exs @@ -0,0 +1,108 @@ +defmodule ExCodecs.DocumentationTest do + use ExUnit.Case, async: true + + @documented_modules [ + ExCodecs, + ExCodecs.Application, + ExCodecs.Codec, + ExCodecs.CodecRegistry, + ExCodecs.Compression, + ExCodecs.Compression.Blosc2, + ExCodecs.Compression.Bzip2, + ExCodecs.Compression.Lz4, + ExCodecs.Compression.Snappy, + ExCodecs.Compression.Zstd, + ExCodecs.Error, + ExCodecs.Native, + ExCodecs.Spatial, + ExCodecs.Spatial.Bounds, + ExCodecs.Spatial.Codec.Binary, + ExCodecs.Spatial.Codec.Gsplat, + ExCodecs.Spatial.Codec.PLY, + ExCodecs.Spatial.Gaussian, + ExCodecs.Spatial.GaussianCloud, + ExCodecs.Spatial.Metadata, + ExCodecs.Spatial.Point, + ExCodecs.Spatial.PointCloud, + ExCodecs.Spatial.Stream, + ExCodecs.Spatial.Transform + ] + + @function_sections ["## Arguments", "## Returns", "## Raises", "## Example"] + + test "all visible public functions document their contract and an example" do + failures = + for module <- @documented_modules, + {{kind, name, arity}, annotation, _signature, doc, metadata} <- docs(module), + kind in [:function, :macro], + not generated?(annotation, metadata), + is_map(doc), + section <- @function_sections, + not String.contains?(doc["en"], section), + do: "#{inspect(module)}.#{name}/#{arity} is missing #{section}" + + assert failures == [], Enum.join(failures, "\n") + end + + test "all visible public types have descriptions and examples" do + failures = + for module <- @documented_modules, + {{:type, name, arity}, _line, _signature, doc, _metadata} <- docs(module), + do: type_doc_failure(module, name, arity, doc) + + failures = Enum.reject(failures, &is_nil/1) + assert failures == [], Enum.join(failures, "\n") + end + + test "all callbacks document a realistic implementation" do + failures = + for module <- @documented_modules, + {{kind, name, arity}, _line, _signature, doc, _metadata} <- docs(module), + kind in [:callback, :macrocallback], + do: callback_doc_failure(module, name, arity, doc) + + failures = Enum.reject(failures, &is_nil/1) + assert failures == [], Enum.join(failures, "\n") + end + + defp docs(module) do + case Code.fetch_docs(module) do + {:docs_v1, _, _, _, _, _, docs} -> docs + {:error, reason} -> flunk("Could not fetch docs for #{inspect(module)}: #{inspect(reason)}") + end + end + + defp generated?(annotation, metadata) do + annotation_generated? = + is_list(annotation) and Keyword.get(annotation, :generated, false) + + metadata_generated? = + (is_map(metadata) and Map.get(metadata, :generated, false)) or + (is_list(metadata) and Keyword.get(metadata, :generated, false)) + + annotation_generated? or metadata_generated? + end + + defp type_doc_failure(module, name, arity, doc) when not is_map(doc), + do: "#{inspect(module)}.#{name}/#{arity} has no @typedoc" + + defp type_doc_failure(module, name, arity, %{"en" => text}) do + if String.contains?(String.downcase(text), "example") do + nil + else + "#{inspect(module)}.#{name}/#{arity} is missing a realistic type example" + end + end + + defp callback_doc_failure(module, name, arity, doc) when not is_map(doc), + do: "#{inspect(module)}.#{name}/#{arity} has no callback documentation" + + defp callback_doc_failure(module, name, arity, %{"en" => text}) do + required = ["## Arguments", "## Returns", "## Raises", "implementation"] + + case Enum.find(required, &(not String.contains?(String.downcase(text), String.downcase(&1)))) do + nil -> nil + missing -> "#{inspect(module)}.#{name}/#{arity} callback is missing #{missing}" + end + end +end diff --git a/test/ex_codecs/nif_test.exs b/test/ex_codecs/nif_test.exs index b20fa31..023daa3 100644 --- a/test/ex_codecs/nif_test.exs +++ b/test/ex_codecs/nif_test.exs @@ -18,6 +18,11 @@ defmodule ExCodecs.NIFTest do NIF.wrap(:lz4, {:error, :decompression_failed}) end + test "wraps output_limit_exceeded error" do + assert {:error, %ExCodecs.Error{reason: :output_limit_exceeded, codec: :zstd}} = + NIF.wrap(:zstd, {:error, :output_limit_exceeded}) + end + test "wraps invalid_data error" do assert {:error, %ExCodecs.Error{reason: :invalid_data, codec: :blosc2}} = NIF.wrap(:blosc2, {:error, :invalid_data}) diff --git a/test/ex_codecs/spatial/bounds_test.exs b/test/ex_codecs/spatial/bounds_test.exs new file mode 100644 index 0000000..1c8c699 --- /dev/null +++ b/test/ex_codecs/spatial/bounds_test.exs @@ -0,0 +1,21 @@ +defmodule ExCodecs.Spatial.BoundsTest do + use ExUnit.Case, async: true + + alias ExCodecs.Spatial.{Bounds, Point} + + test "computes bounds from points" do + points = [Point.new(0, 0, 0), Point.new(2, 4, 6)] + bounds = Bounds.from_points(points) + + assert bounds.min_x == 0.0 + assert bounds.max_z == 6.0 + assert Bounds.center(bounds) == {1.0, 2.0, 3.0} + assert Bounds.size(bounds) == {2.0, 4.0, 6.0} + assert Bounds.contains?(bounds, {1.0, 1.0, 1.0}) + refute Bounds.contains?(bounds, {10.0, 0.0, 0.0}) + end + + test "empty enumerable yields nil" do + assert Bounds.from_points([]) == nil + end +end diff --git a/test/ex_codecs/spatial/codec/binary_test.exs b/test/ex_codecs/spatial/codec/binary_test.exs new file mode 100644 index 0000000..aa9788c --- /dev/null +++ b/test/ex_codecs/spatial/codec/binary_test.exs @@ -0,0 +1,33 @@ +defmodule ExCodecs.Spatial.Codec.BinaryTest do + use ExUnit.Case, async: true + + alias ExCodecs.Spatial.Codec.Binary + alias ExCodecs.Spatial.{Point, PointCloud} + + test "round-trips colored points with normals" do + cloud = + PointCloud.new([ + Point.new(1.0, 2.0, 3.0, color: {10, 20, 30}, normal: {0.0, 1.0, 0.0}), + Point.new(4.0, 5.0, 6.0, color: {40, 50, 60}, normal: {1.0, 0.0, 0.0}) + ]) + + assert {:ok, bin} = Binary.encode(cloud) + assert String.starts_with?(bin, "EXCP") + assert {:ok, decoded} = Binary.decode(bin) + assert length(decoded.points) == 2 + assert hd(decoded.points).color == {10, 20, 30} + assert hd(decoded.points).normal == {0.0, 1.0, 0.0} + end + + test "round-trips RGBA" do + cloud = PointCloud.new([Point.new(0, 0, 0, color: {1, 2, 3, 4})]) + assert {:ok, bin} = Binary.encode(cloud) + assert {:ok, decoded} = Binary.decode(bin) + assert hd(decoded.points).color == {1, 2, 3, 4} + end + + test "rejects bad magic" do + assert {:error, %{reason: :invalid_data}} = + Binary.decode(<<"XXXX", 0, 1, 0, 0, 0, 0, 0, 0, 0, 0>>) + end +end diff --git a/test/ex_codecs/spatial/codec/gsplat_test.exs b/test/ex_codecs/spatial/codec/gsplat_test.exs new file mode 100644 index 0000000..15e38d9 --- /dev/null +++ b/test/ex_codecs/spatial/codec/gsplat_test.exs @@ -0,0 +1,41 @@ +defmodule ExCodecs.Spatial.Codec.GsplatTest do + use ExUnit.Case, async: true + + alias ExCodecs.Spatial.Codec.Gsplat + alias ExCodecs.Spatial.{Gaussian, GaussianCloud} + + test "round-trips Gaussians" do + cloud = + GaussianCloud.new([ + Gaussian.new({1.0, 2.0, 3.0}, + color: {0.1, 0.2, 0.3}, + opacity: 0.9, + scale: {0.2, 0.3, 0.4}, + rotation: {0.9, 0.1, 0.0, 0.0} + ) + ]) + + assert {:ok, bin} = Gsplat.encode(cloud) + assert String.starts_with?(bin, "GSPL") + assert {:ok, decoded} = Gsplat.decode(bin) + [g] = decoded.gaussians + assert_in_delta elem(g.position, 2), 3.0, 0.0001 + assert_in_delta g.opacity, 0.9, 0.0001 + end + + test "round-trips SH rest coefficients" do + cloud = + GaussianCloud.new([ + Gaussian.new({0.0, 0.0, 0.0}, + color: {0.5, 0.5, 0.5}, + sh: [[0.5, 0.5, 0.5], [0.1, 0.2, 0.3], [0.4, 0.5, 0.6]] + ) + ]) + + assert {:ok, bin} = Gsplat.encode(cloud) + assert {:ok, decoded} = Gsplat.decode(bin) + [g] = decoded.gaussians + assert is_list(g.sh) + assert length(List.flatten(tl(g.sh))) == 6 + end +end diff --git a/test/ex_codecs/spatial/codec/ply_extra_test.exs b/test/ex_codecs/spatial/codec/ply_extra_test.exs new file mode 100644 index 0000000..ec6c119 --- /dev/null +++ b/test/ex_codecs/spatial/codec/ply_extra_test.exs @@ -0,0 +1,72 @@ +defmodule ExCodecs.Spatial.Codec.PLYExtraTest do + use ExUnit.Case, async: true + + alias ExCodecs.Spatial.Codec.PLY + alias ExCodecs.Spatial.{Gaussian, GaussianCloud, Point, PointCloud} + + test "binary big-endian round-trip" do + cloud = PointCloud.new([Point.new(1.5, 2.5, 3.5, color: {9, 8, 7})]) + assert {:ok, bin} = PLY.encode(cloud, format: :binary_be) + assert bin =~ "binary_big_endian" + assert {:ok, decoded} = PLY.decode(bin) + assert_in_delta hd(decoded.points).x, 1.5, 0.0001 + assert hd(decoded.points).color == {9, 8, 7} + end + + test "Gaussian binary PLY with SH" do + cloud = + GaussianCloud.new([ + Gaussian.new({0.0, 1.0, 2.0}, + color: {0.1, 0.2, 0.3}, + sh: [[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]] + ) + ]) + + assert {:ok, bin} = PLY.encode(cloud, format: :binary) + assert {:ok, decoded} = PLY.decode(bin) + assert %GaussianCloud{} = decoded + assert hd(decoded.gaussians).sh != nil + end + + test "encode rejects non-cloud" do + assert {:error, %{reason: :invalid_data}} = PLY.encode(:nope) + end + + test "stream_decode from binary and missing file path content" do + cloud = PointCloud.new([Point.new(0.0, 0.0, 1.0)]) + {:ok, bin} = PLY.encode(cloud) + + points = PLY.stream_decode(bin) |> Enum.to_list() + assert length(points) == 1 + + # Non-existent path-like string without separators is treated as binary PLY data + bad = PLY.stream_decode("not a ply file at all") |> Enum.to_list() + assert [{:error, %{reason: :invalid_data}}] = bad + end + + test "truncated vertex body" do + header = """ + ply + format ascii 1.0 + element vertex 2 + property float x + property float y + property float z + end_header + 0 0 0 + """ + + assert {:error, %{reason: :invalid_data}} = PLY.decode(header) + end + + test "unsupported format line" do + data = """ + ply + format weird 1.0 + element vertex 0 + end_header + """ + + assert {:error, %{reason: :invalid_data}} = PLY.decode(data) + end +end diff --git a/test/ex_codecs/spatial/codec/ply_test.exs b/test/ex_codecs/spatial/codec/ply_test.exs new file mode 100644 index 0000000..88c2b30 --- /dev/null +++ b/test/ex_codecs/spatial/codec/ply_test.exs @@ -0,0 +1,95 @@ +defmodule ExCodecs.Spatial.Codec.PLYTest do + use ExUnit.Case, async: true + + alias ExCodecs.Spatial.Codec.PLY + alias ExCodecs.Spatial.{Gaussian, GaussianCloud, Point, PointCloud} + + describe "point cloud ASCII PLY" do + test "round-trips XYZRGB" do + cloud = + PointCloud.new([ + Point.new(1.0, 2.0, 3.0, color: {255, 128, 0}), + Point.new(-1.0, 0.5, 9.0, color: {0, 255, 64}) + ]) + + assert {:ok, binary} = PLY.encode(cloud, format: :ascii) + assert String.starts_with?(binary, "ply\n") + assert binary =~ "format ascii 1.0" + assert binary =~ "property uchar red" + + assert {:ok, decoded} = PLY.decode(binary) + assert length(decoded.points) == 2 + assert hd(decoded.points).color == {255, 128, 0} + assert Float.round(hd(decoded.points).x, 5) == 1.0 + end + + test "round-trips XYZ + normal + attributes" do + cloud = + PointCloud.new([ + Point.new(0.0, 1.0, 2.0, + normal: {0.0, 1.0, 0.0}, + attributes: %{"intensity" => 0.75} + ) + ]) + + assert {:ok, binary} = PLY.encode(cloud) + assert {:ok, decoded} = PLY.decode(binary) + [p] = decoded.points + assert p.normal == {0.0, 1.0, 0.0} + assert_in_delta p.attributes["intensity"], 0.75, 0.0001 + end + + test "round-trips binary little-endian PLY" do + cloud = + PointCloud.new([ + Point.new(1.25, 2.5, 3.75, color: {1, 2, 3, 4}) + ]) + + assert {:ok, binary} = PLY.encode(cloud, format: :binary) + assert binary =~ "binary_little_endian" + assert {:ok, decoded} = PLY.decode(binary) + [p] = decoded.points + assert_in_delta p.x, 1.25, 0.0001 + assert p.color == {1, 2, 3, 4} + end + end + + describe "Gaussian PLY" do + test "round-trips Gaussian clouds" do + cloud = + GaussianCloud.new([ + Gaussian.new({1.0, 2.0, 3.0}, + color: {0.1, 0.2, 0.3}, + opacity: 0.8, + scale: {0.5, 0.6, 0.7}, + rotation: {1.0, 0.0, 0.0, 0.0} + ) + ]) + + assert {:ok, binary} = PLY.encode(cloud, format: :ascii) + assert binary =~ "f_dc_0" + assert binary =~ "opacity" + + assert {:ok, decoded} = PLY.decode(binary) + assert %GaussianCloud{} = decoded + [g] = decoded.gaussians + assert_in_delta elem(g.position, 0), 1.0, 0.0001 + assert_in_delta elem(g.color, 1), 0.2, 0.0001 + assert_in_delta g.opacity, 0.8, 0.0001 + end + + test "can force point_cloud interpretation" do + cloud = + GaussianCloud.new([ + Gaussian.new({0.0, 0.0, 0.0}, color: {0.5, 0.5, 0.5}) + ]) + + assert {:ok, binary} = PLY.encode(cloud) + assert {:ok, %PointCloud{}} = PLY.decode(binary, as: :point_cloud) + end + end + + test "rejects non-PLY data" do + assert {:error, %{reason: :invalid_data}} = PLY.decode("not a ply file") + end +end diff --git a/test/ex_codecs/spatial/coverage_test.exs b/test/ex_codecs/spatial/coverage_test.exs new file mode 100644 index 0000000..086d643 --- /dev/null +++ b/test/ex_codecs/spatial/coverage_test.exs @@ -0,0 +1,234 @@ +defmodule ExCodecs.Spatial.CoverageTest do + use ExUnit.Case, async: true + + alias ExCodecs.Spatial + alias ExCodecs.Spatial.{Bounds, Gaussian, GaussianCloud, Metadata, Point, PointCloud} + alias ExCodecs.Spatial.Codec.{Binary, Gsplat, PLY} + alias ExCodecs.Spatial.Stream, as: SpatialStream + + describe "binary codec edges" do + test "xyz-only round-trip" do + cloud = PointCloud.new([Point.new(1.0, 2.0, 3.0), Point.new(4.0, 5.0, 6.0)]) + assert {:ok, bin} = Binary.encode(cloud) + assert {:ok, decoded} = Binary.decode(bin) + assert Enum.map(decoded.points, & &1.color) == [nil, nil] + end + + test "unsupported version and truncation" do + assert {:error, _} = Binary.decode(<<"EXCP", 99::little-16, 0::little-16, 1::little-64>>) + assert {:error, _} = Binary.decode(<<"EXCP", 1::little-16, 1::little-16, 1::little-64, 0, 1>>) + + assert [{:error, _}] = + Binary.stream_decode(<<"nope">>) |> Enum.to_list() + end + + test "encode rejects non-cloud" do + assert {:error, %{codec: :spatial_binary}} = Binary.encode(%{}) + end + end + + describe "gsplat edges" do + test "unsupported version and truncation" do + assert {:error, _} = + Gsplat.decode(<<"GSPL", 9::little-16, 0::little-16, 1::little-64, 0::little-16>>) + + assert {:error, _} = + Gsplat.decode( + <<"GSPL", 1::little-16, 0::little-16, 1::little-64, 0::little-16, 1, 2>> + ) + + assert [{:error, _}] = Gsplat.stream_decode(<<"nope">>) |> Enum.to_list() + assert {:error, _} = Gsplat.encode(%{}) + end + end + + describe "PLY typed properties" do + test "ascii double / int / uchar mix" do + data = """ + ply + format ascii 1.0 + comment typed + element vertex 1 + property double x + property double y + property double z + property int label + property ushort code + property short delta + property uint big + property char tiny + end_header + 1.0 2.0 3.0 7 100 -3 999 -1 + """ + + assert {:ok, cloud} = PLY.decode(data) + [p] = cloud.points + assert_in_delta p.x, 1.0, 0.0001 + assert p.attributes["label"] == 7 + end + + test "binary_le with mixed numeric types" do + # Build via encode then is float/uchar; craft header+body manually for doubles + x = 1.25 + y = 2.5 + z = 3.75 + label = 42 + + header = """ + ply + format binary_little_endian 1.0 + element vertex 1 + property float x + property float y + property float z + property int label + end_header + """ + + body = + <> + + assert {:ok, cloud} = PLY.decode(header <> body) + assert hd(cloud.points).attributes["label"] == 42 + end + + test "binary_be floats" do + header = """ + ply + format binary_big_endian 1.0 + element vertex 1 + property float x + property float y + property float z + property ushort code + property uchar red + property uchar green + property uchar blue + end_header + """ + + body = + <<1.0::big-float-32, 2.0::big-float-32, 3.0::big-float-32, 7::big-unsigned-16, 10, 20, 30>> + + assert {:ok, cloud} = PLY.decode(header <> body) + assert hd(cloud.points).color == {10, 20, 30} + assert hd(cloud.points).attributes["code"] == 7 + end + + test "missing magic / format / vertex element" do + assert {:error, _} = PLY.decode("format ascii 1.0\nend_header\n") + assert {:error, _} = PLY.decode("ply\nelement vertex 0\nend_header\n") + + assert {:error, _} = + PLY.decode("ply\nformat ascii 1.0\nend_header\n") + end + + test "list properties unsupported" do + data = """ + ply + format ascii 1.0 + element vertex 0 + property list uchar int vertex_indices + end_header + """ + + assert {:error, _} = PLY.decode(data) + end + + test "stream_decode file path" do + path = Path.join(System.tmp_dir!(), "ex_codecs_ply_#{System.unique_integer([:positive])}.ply") + on_exit(fn -> File.rm(path) end) + + cloud = PointCloud.new([Point.new(9.0, 8.0, 7.0)]) + {:ok, bin} = PLY.encode(cloud) + File.write!(path, bin) + + assert [%Point{x: x}] = PLY.stream_decode(path) |> Enum.to_list() + assert_in_delta x, 9.0, 0.0001 + end + + test "Gaussian ascii with f_rest" do + cloud = + GaussianCloud.new([ + Gaussian.new({0, 0, 0}, sh: [[0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9]]) + ]) + + assert {:ok, bin} = PLY.encode(cloud, format: :ascii, comments: ["sh test"]) + assert bin =~ "f_rest_0" + assert {:ok, decoded} = PLY.decode(bin, as: :gaussian_cloud) + assert hd(decoded.gaussians).sh != nil + end + end + + describe "stream helpers" do + test "encode_to_file with enumerable and bad path" do + path = Path.join(System.tmp_dir!(), "ex_codecs_pts_#{System.unique_integer([:positive])}.ply") + on_exit(fn -> File.rm(path) end) + + assert :ok = + SpatialStream.encode_to_file([Point.new(1, 2, 3)], path, format: :ply) + + assert {:error, _} = + SpatialStream.encode_to_file([Point.new(1, 2, 3)], "/no/such/dir/x.ply", + format: :ply + ) + end + + test "stream_decode reads spatial_binary file" do + path = + Path.join(System.tmp_dir!(), "ex_codecs_bin_#{System.unique_integer([:positive])}.excp") + + on_exit(fn -> File.rm(path) end) + + {:ok, bin} = Spatial.encode(PointCloud.new([Point.new(1, 2, 3)]), format: :spatial_binary) + File.write!(path, bin) + + assert [%Point{}] = + Spatial.stream_decode(path, format: :spatial_binary) |> Enum.to_list() + end + + test "stream encode error tuples" do + assert {:error, _} = + Spatial.stream_encode([{:error, :boom}], format: :ply) + end + + test "unsupported empty format" do + assert {:error, %{reason: :unsupported_codec}} = + Spatial.stream_encode([], format: :sog) + end + end + + describe "point cloud remaining branches" do + test "explicit bounds and mixed color detection" do + bounds = Bounds.new({0, 0, 0}, {1, 1, 1}) + + cloud = + PointCloud.new([Point.new(0, 0, 0, color: {1, 2, 3}), Point.new(1, 1, 1)], + bounds: bounds, + metadata: Metadata.new(comments: ["x"]) + ) + + assert cloud.bounds == bounds + refute PointCloud.colored?(cloud) + refute PointCloud.has_normals?(cloud) + end + end + + describe "encode_to_file extras" do + test "gaussian cloud and gsplat file stream" do + path = Path.join(System.tmp_dir!(), "ex_codecs_g_#{System.unique_integer([:positive])}.gspl") + on_exit(fn -> File.rm(path) end) + + cloud = GaussianCloud.new([Gaussian.new({1.0, 2.0, 3.0})]) + assert :ok = SpatialStream.encode_to_file(cloud, path, format: :gsplat) + + assert [%Gaussian{}] = + Spatial.stream_decode(path, format: :gsplat) |> Enum.to_list() + end + + test "propagates encode errors" do + assert {:error, _} = + SpatialStream.encode_to_file([Point.new(0, 0, 0)], "/tmp/x.gspl", format: :gsplat) + end + end +end diff --git a/test/ex_codecs/spatial/point_test.exs b/test/ex_codecs/spatial/point_test.exs new file mode 100644 index 0000000..5f5886b --- /dev/null +++ b/test/ex_codecs/spatial/point_test.exs @@ -0,0 +1,26 @@ +defmodule ExCodecs.Spatial.PointTest do + use ExUnit.Case, async: true + + alias ExCodecs.Spatial.Point + + doctest ExCodecs.Spatial.Point + + test "creates points with optional color and normal" do + p = + Point.new(1, 2, 3, + color: {10, 20, 30, 255}, + normal: {0.0, 1.0, 0.0}, + attributes: %{"intensity" => 0.5} + ) + + assert Point.coords(p) == {1.0, 2.0, 3.0} + assert Point.colored?(p) + assert Point.has_normal?(p) + assert p.attributes["intensity"] == 0.5 + end + + test "normalizes atom attribute keys to strings" do + p = Point.new(0, 0, 0, attributes: %{:intensity => 0.25, "label" => "a"}) + assert p.attributes == %{"intensity" => 0.25, "label" => "a"} + end +end diff --git a/test/ex_codecs/spatial/stream_test.exs b/test/ex_codecs/spatial/stream_test.exs new file mode 100644 index 0000000..879479f --- /dev/null +++ b/test/ex_codecs/spatial/stream_test.exs @@ -0,0 +1,69 @@ +defmodule ExCodecs.Spatial.StreamTest do + use ExUnit.Case, async: true + + alias ExCodecs.Spatial + alias ExCodecs.Spatial.{Gaussian, Point, PointCloud} + + test "stream encode empty clouds" do + assert {:ok, ply} = Spatial.stream_encode([], format: :ply) + assert {:ok, %PointCloud{points: []}} = Spatial.decode(ply, format: :ply) + + assert {:ok, bin} = Spatial.stream_encode([], format: :spatial_binary) + assert {:ok, %PointCloud{points: []}} = Spatial.decode(bin, format: :spatial_binary) + + assert {:ok, gspl} = Spatial.stream_encode([], format: :gsplat) + assert {:ok, %{gaussians: []}} = Spatial.decode(gspl, format: :gsplat) + end + + test "stream decode unsupported format yields error" do + assert [{:error, %{reason: :unsupported_codec}}] = + Spatial.stream_decode(<<>>, format: :sog) |> Enum.to_list() + end + + test "stream encode rejects non-spatial items" do + assert {:error, %{reason: :invalid_data}} = + Spatial.stream_encode([:nope], format: :ply) + end + + test "stream decode spatial_binary and gsplat binaries" do + cloud = PointCloud.new([Point.new(1.0, 2.0, 3.0)]) + {:ok, bin} = Spatial.encode(cloud, format: :spatial_binary) + + points = Spatial.stream_decode(bin, format: :spatial_binary) |> Enum.to_list() + assert length(points) == 1 + + gs = [Gaussian.new({0.0, 0.0, 0.0})] + {:ok, gbin} = Spatial.stream_encode(gs, format: :gsplat) + items = Spatial.stream_decode(gbin, format: :gsplat) |> Enum.to_list() + assert length(items) == 1 + end + + test "encode rejects wrong type for format" do + cloud = PointCloud.new([Point.new(0, 0, 0)]) + assert {:error, %{codec: :gsplat}} = Spatial.encode(cloud, format: :gsplat) + + gcloud = Spatial.GaussianCloud.new([Gaussian.new({0, 0, 0})]) + + assert {:error, %{codec: :spatial_binary}} = + Spatial.encode(gcloud, format: :spatial_binary) + + assert {:error, %{reason: :unsupported_codec}} = + Spatial.encode(cloud, format: :unknown) + + assert {:error, %{reason: :invalid_data}} = Spatial.encode(:nope, format: :ply) + assert {:error, %{reason: :invalid_data}} = Spatial.decode(123, format: :ply) + end + + test "registry decode with format: points at Spatial category" do + assert {:error, %{reason: :invalid_options, message: message}} = + ExCodecs.decode(<<"ply\n">>, format: :ply) + + assert message =~ "ExCodecs.Spatial.decode" + end + + test "top-level stream helpers still delegate to Spatial" do + cloud = PointCloud.new([Point.new(1.0, 2.0, 3.0)]) + {:ok, bin} = Spatial.encode(cloud, format: :ply) + assert [%Point{}] = ExCodecs.stream_decode(bin, format: :ply) |> Enum.to_list() + end +end diff --git a/test/ex_codecs/spatial/types_test.exs b/test/ex_codecs/spatial/types_test.exs new file mode 100644 index 0000000..7d821a9 --- /dev/null +++ b/test/ex_codecs/spatial/types_test.exs @@ -0,0 +1,74 @@ +defmodule ExCodecs.Spatial.TypesTest do + use ExUnit.Case, async: true + + alias ExCodecs.Spatial.{ + Bounds, + Gaussian, + GaussianCloud, + Metadata, + Point, + PointCloud, + Transform + } + + describe "PointCloud" do + test "size, colored?, normals, add_point, with_bounds" do + empty = PointCloud.new([]) + assert PointCloud.size(empty) == 0 + refute PointCloud.colored?(empty) + refute PointCloud.has_normals?(empty) + + p1 = Point.new(0, 0, 0, color: {1, 2, 3}, normal: {0.0, 1.0, 0.0}) + p2 = Point.new(1, 1, 1, color: {4, 5, 6}, normal: {1.0, 0.0, 0.0}) + cloud = PointCloud.new([p1], compute_bounds: false) + assert cloud.bounds == nil + + cloud = PointCloud.add_point(cloud, p2) + assert PointCloud.size(cloud) == 2 + assert PointCloud.colored?(cloud) + assert PointCloud.has_normals?(cloud) + assert cloud.bounds.max_x == 1.0 + + cloud = PointCloud.with_bounds(%{cloud | bounds: nil}) + assert %Bounds{} = cloud.bounds + end + end + + describe "GaussianCloud" do + test "size and with_bounds" do + cloud = GaussianCloud.new([Gaussian.new({1, 2, 3})], compute_bounds: false) + assert GaussianCloud.size(cloud) == 1 + assert cloud.bounds == nil + cloud = GaussianCloud.with_bounds(cloud) + assert cloud.bounds.min_x == 1.0 + end + end + + describe "Metadata" do + test "put, get, add_comment" do + meta = Metadata.new(source: "test") + meta = Metadata.put(meta, "crs", "EPSG:4326") + meta = Metadata.add_comment(meta, "hello") + assert Metadata.get(meta, "crs") == "EPSG:4326" + assert Metadata.get(meta, "missing", :default) == :default + assert meta.comments == ["hello"] + assert meta.source == "test" + end + end + + describe "Transform" do + test "identity and new" do + assert Transform.identity().scale == 1.0 + + t = + Transform.new( + translation: {1, 2, 3}, + rotation: {0.5, 0.5, 0.5, 0.5}, + scale: 2 + ) + + assert t.translation == {1.0, 2.0, 3.0} + assert t.scale == 2.0 + end + end +end diff --git a/test/ex_codecs/spatial_test.exs b/test/ex_codecs/spatial_test.exs new file mode 100644 index 0000000..a0dba03 --- /dev/null +++ b/test/ex_codecs/spatial_test.exs @@ -0,0 +1,136 @@ +defmodule ExCodecs.SpatialTest do + use ExUnit.Case, async: true + + alias ExCodecs.Spatial + alias ExCodecs.Spatial.{Gaussian, GaussianCloud, Point, PointCloud} + + defmodule CatalogSpatialCodec do + def encode(%PointCloud{}, []), do: {:ok, "catalog-spatial"} + def decode("catalog-spatial", []), do: {:ok, PointCloud.new([])} + end + + describe "available_formats/0" do + test "lists supported formats" do + assert Spatial.available_formats() == [:ply, :spatial_binary, :gsplat] + assert :ply in Spatial.available_formats() + assert :gsplat in Spatial.available_formats() + assert Spatial.supports?(:ply) + refute Spatial.supports?(:sog) + end + + test "spatial formats participate in the shared catalog" do + assert :ply in ExCodecs.available_codecs() + assert ExCodecs.available_codecs(:spatial) == [:gsplat, :ply, :spatial_binary] + assert ExCodecs.supports?(:ply) + + assert {:ok, info} = ExCodecs.codec_info(:ply) + assert info.category == :spatial + assert info.interface == :spatial + end + + test "registered spatial extensions are discovered and dispatched" do + format = :"spatial_test_#{System.unique_integer([:positive])}" + + on_exit(fn -> + ExCodecs.CodecRegistry.unregister(format) + end) + + assert :ok = + ExCodecs.CodecRegistry.register( + format, + CatalogSpatialCodec, + :spatial, + :spatial + ) + + assert format in Spatial.available_formats() + assert Spatial.supports?(format) + + assert {:ok, "catalog-spatial"} = + Spatial.encode(PointCloud.new([]), format: format) + + assert {:ok, %PointCloud{points: []}} = + Spatial.decode("catalog-spatial", format: format) + end + + test "an unavailable spatial catalog entry returns codec_unavailable" do + format = :"unavailable_spatial_test_#{System.unique_integer([:positive])}" + + on_exit(fn -> + ExCodecs.CodecRegistry.unregister(format) + end) + + assert :ok = + ExCodecs.CodecRegistry.register_unavailable(format, :spatial, :spatial) + + refute Spatial.supports?(format) + + assert {:error, %ExCodecs.Error{reason: :codec_unavailable, codec: ^format}} = + Spatial.decode("payload", format: format) + end + end + + describe "encode/decode" do + test "PLY via Spatial API" do + cloud = PointCloud.new([Point.new(1.0, 2.0, 3.0)]) + assert {:ok, bin} = Spatial.encode(cloud, format: :ply) + assert {:ok, decoded} = Spatial.decode(bin, format: :ply) + assert length(decoded.points) == 1 + end + + test "registry API rejects spatial structs and points at Spatial" do + cloud = PointCloud.new([Point.new(0.0, 1.0, 2.0, color: {9, 8, 7})]) + + assert {:error, %ExCodecs.Error{reason: :invalid_data}} = + ExCodecs.encode(cloud, format: :ply) + + assert {:ok, bin} = Spatial.encode(cloud, format: :ply) + + assert {:error, %ExCodecs.Error{reason: :invalid_options}} = + ExCodecs.decode(bin, format: :ply) + + assert {:ok, decoded} = Spatial.decode(bin, format: :ply) + assert hd(decoded.points).color == {9, 8, 7} + + assert {:error, %ExCodecs.Error{reason: :invalid_options}} = + ExCodecs.encode(:ply, bin) + + assert {:error, %ExCodecs.Error{reason: :invalid_options}} = + ExCodecs.decode(:ply, bin) + end + + test "stream decode and encode" do + cloud = + PointCloud.new([ + Point.new(0.0, 0.0, 0.0), + Point.new(1.0, 1.0, 1.0) + ]) + + assert {:ok, bin} = Spatial.encode(cloud, format: :ply) + + points = + Spatial.stream_decode(bin, format: :ply) + |> Enum.to_list() + + assert length(points) == 2 + assert {:ok, bin2} = Spatial.stream_encode(points, format: :ply) + assert {:ok, _} = Spatial.decode(bin2, format: :ply) + end + + test "stream_encode Gaussian cloud via gsplat" do + gs = [Gaussian.new({1.0, 2.0, 3.0})] + assert {:ok, bin} = ExCodecs.stream_encode(gs, format: :gsplat) + assert {:ok, %GaussianCloud{}} = Spatial.decode(bin, format: :gsplat) + end + end + + test "loads example PLY fixtures" do + path = Path.join([:code.priv_dir(:ex_codecs), "examples", "spatial", "cube_corners.ply"]) + assert {:ok, cloud} = Spatial.decode(File.read!(path), format: :ply) + assert length(cloud.points) == 4 + + gpath = Path.join([:code.priv_dir(:ex_codecs), "examples", "spatial", "two_gaussians.ply"]) + assert {:ok, %GaussianCloud{gaussians: gs}} = Spatial.decode(File.read!(gpath), format: :ply) + assert length(gs) == 2 + end +end diff --git a/test/ex_codecs_test.exs b/test/ex_codecs_test.exs index d62175c..77dd088 100644 --- a/test/ex_codecs_test.exs +++ b/test/ex_codecs_test.exs @@ -191,17 +191,21 @@ defmodule ExCodecsTest do describe "codec_unavailable path" do test "encode with unavailable codec returns codec_unavailable error" do - :ok = ExCodecs.CodecRegistry.register_unavailable(:future_codec, :compression) + name = :"future_codec_#{System.unique_integer([:positive])}" + :ok = ExCodecs.CodecRegistry.register_unavailable(name, :compression) + on_exit(fn -> ExCodecs.CodecRegistry.unregister(name) end) assert {:error, %ExCodecs.Error{reason: :codec_unavailable}} = - ExCodecs.encode(:future_codec, "data") + ExCodecs.encode(name, "data") end test "decode with unavailable codec returns codec_unavailable error" do - :ok = ExCodecs.CodecRegistry.register_unavailable(:future_codec2, :compression) + name = :"future_codec2_#{System.unique_integer([:positive])}" + :ok = ExCodecs.CodecRegistry.register_unavailable(name, :compression) + on_exit(fn -> ExCodecs.CodecRegistry.unregister(name) end) assert {:error, %ExCodecs.Error{reason: :codec_unavailable}} = - ExCodecs.decode(:future_codec2, <<1, 2, 3>>) + ExCodecs.decode(name, <<1, 2, 3>>) end end diff --git a/test/fixtures/blosc2/README.md b/test/fixtures/blosc2/README.md new file mode 100644 index 0000000..49d81ed --- /dev/null +++ b/test/fixtures/blosc2/README.md @@ -0,0 +1,11 @@ +# Blosc2 golden chunks + +Generated with python-blosc2 for cross-implementation tests. + +```sh +python3 -c 'import blosc2; print(blosc2.__version__)' +# regenerate with the script in this directory if present, or: +# see ExCodecs.Compression.Blosc2InteropTest +``` + +Files: `*.bin` = compressed chunk, `*.src` = original payload. diff --git a/test/fixtures/blosc2/blosclz_noshuffle.bin b/test/fixtures/blosc2/blosclz_noshuffle.bin new file mode 100644 index 0000000..4bcc81e Binary files /dev/null and b/test/fixtures/blosc2/blosclz_noshuffle.bin differ diff --git a/test/fixtures/blosc2/blosclz_noshuffle.src b/test/fixtures/blosc2/blosclz_noshuffle.src new file mode 100644 index 0000000..2545ade Binary files /dev/null and b/test/fixtures/blosc2/blosclz_noshuffle.src differ diff --git a/test/fixtures/blosc2/lz4_bitshuffle_t8.bin b/test/fixtures/blosc2/lz4_bitshuffle_t8.bin new file mode 100644 index 0000000..1a8f5bf Binary files /dev/null and b/test/fixtures/blosc2/lz4_bitshuffle_t8.bin differ diff --git a/test/fixtures/blosc2/lz4_bitshuffle_t8.src b/test/fixtures/blosc2/lz4_bitshuffle_t8.src new file mode 100644 index 0000000..56114e3 Binary files /dev/null and b/test/fixtures/blosc2/lz4_bitshuffle_t8.src differ diff --git a/test/fixtures/blosc2/lz4_noshuffle_t1.bin b/test/fixtures/blosc2/lz4_noshuffle_t1.bin new file mode 100644 index 0000000..32a6f1b Binary files /dev/null and b/test/fixtures/blosc2/lz4_noshuffle_t1.bin differ diff --git a/test/fixtures/blosc2/lz4_noshuffle_t1.src b/test/fixtures/blosc2/lz4_noshuffle_t1.src new file mode 100644 index 0000000..2545ade Binary files /dev/null and b/test/fixtures/blosc2/lz4_noshuffle_t1.src differ diff --git a/test/fixtures/blosc2/lz4_shuffle_t8.bin b/test/fixtures/blosc2/lz4_shuffle_t8.bin new file mode 100644 index 0000000..1519bb7 Binary files /dev/null and b/test/fixtures/blosc2/lz4_shuffle_t8.bin differ diff --git a/test/fixtures/blosc2/lz4_shuffle_t8.src b/test/fixtures/blosc2/lz4_shuffle_t8.src new file mode 100644 index 0000000..56114e3 Binary files /dev/null and b/test/fixtures/blosc2/lz4_shuffle_t8.src differ diff --git a/test/fixtures/blosc2/lz4hc_noshuffle.bin b/test/fixtures/blosc2/lz4hc_noshuffle.bin new file mode 100644 index 0000000..e9b3e5a Binary files /dev/null and b/test/fixtures/blosc2/lz4hc_noshuffle.bin differ diff --git a/test/fixtures/blosc2/lz4hc_noshuffle.src b/test/fixtures/blosc2/lz4hc_noshuffle.src new file mode 100644 index 0000000..2545ade Binary files /dev/null and b/test/fixtures/blosc2/lz4hc_noshuffle.src differ diff --git a/test/fixtures/blosc2/zlib_noshuffle.bin b/test/fixtures/blosc2/zlib_noshuffle.bin new file mode 100644 index 0000000..ffe3a63 Binary files /dev/null and b/test/fixtures/blosc2/zlib_noshuffle.bin differ diff --git a/test/fixtures/blosc2/zlib_noshuffle.src b/test/fixtures/blosc2/zlib_noshuffle.src new file mode 100644 index 0000000..2545ade Binary files /dev/null and b/test/fixtures/blosc2/zlib_noshuffle.src differ diff --git a/test/fixtures/blosc2/zstd_shuffle_t8.bin b/test/fixtures/blosc2/zstd_shuffle_t8.bin new file mode 100644 index 0000000..dae979c Binary files /dev/null and b/test/fixtures/blosc2/zstd_shuffle_t8.bin differ diff --git a/test/fixtures/blosc2/zstd_shuffle_t8.src b/test/fixtures/blosc2/zstd_shuffle_t8.src new file mode 100644 index 0000000..56114e3 Binary files /dev/null and b/test/fixtures/blosc2/zstd_shuffle_t8.src differ