ExCodecs

An extensible BEAM-native codec framework for Elixir with specialized category APIs. Binary registry codecs use ExCodecs.encode/3 / decode/3; ExCodecs.Spatial handles point-cloud and Gaussian domain types.

Blosc2 produces C-Blosc2-compatible chunks only (not super-chunk / B2ND / .b2frame). Standalone :snappy is separate from Blosc2 cname:.

Design Philosophy

ExCodecs is not a compression library. It is a codec framework.

Compression and spatial formats are categories in one framework. Each category uses an API shaped for its data:

• Binary→binary registry codecs implement ExCodecs.Codec and use ExCodecs.encode/3 / decode/3.
• Spatial struct↔format codecs use ExCodecs.Spatial.

Discovery is category-specific: ExCodecs.available_codecs/0 lists registered binary codecs, while ExCodecs.Spatial.available_formats/0 lists spatial formats. This keeps one framework and error model without forcing unlike data shapes through an overloaded function.

The encode/decode naming is category-agnostic: for compression codecs, encoding is compressing and decoding is decompressing; for a future hash codec, encoding would produce a digest and decoding would verify it.

Installation

Add ex_codecs to your list of dependencies in mix.exs:

def deps do
  [
    {:ex_codecs, "~> 0.2.3"}
  ]
end

Then fetch dependencies and compile:

mix deps.get && mix compile

Precompiled NIF binaries are available for macOS (Intel and ARM64), Linux (x86_64 and ARM64, glibc and musl), and Windows (x86_64). They are downloaded automatically from the GitHub releases when you run mix deps.get. If a precompiled artifact is not available for your target, ExCodecs falls back to compiling the Rust NIF from source (requires Rust 1.94+). The native crate is pure Rust (no C toolchain / system compression libraries).

Quick Start

# Registry codecs (primary API — always codec atom + binary)
{:ok, compressed} = ExCodecs.encode(:zstd, "hello world")
{:ok, original}  = ExCodecs.decode(:zstd, compressed)
original #=> "hello world"

{:ok, compressed} = ExCodecs.encode(:zstd, my_binary, level: 9)
{:ok, compressed} = ExCodecs.encode(:blosc2, my_binary, cname: :zstd, clevel: 5, shuffle: :byte)

# Category alias for compression
{:ok, compressed} = ExCodecs.Compression.compress(:lz4, data)
{:ok, original}   = ExCodecs.Compression.decompress(:lz4, compressed)

# Shared discovery
ExCodecs.available_codecs()  #=> [:blosc2, :bzip2, :gsplat, :lz4, :ply, :snappy, :spatial_binary, :zstd]
ExCodecs.available_codecs(:compression) #=> [:blosc2, :bzip2, :lz4, :snappy, :zstd]
ExCodecs.available_codecs(:spatial)     #=> [:gsplat, :ply, :spatial_binary]
ExCodecs.supports?(:zstd)    #=> true
ExCodecs.codec_info(:zstd)   #=> {:ok, %ExCodecs.Codec{name: :zstd, category: :compression, ...}}

# Spatial category API (structs ↔ formats)
alias ExCodecs.Spatial.{Point, PointCloud}
cloud = PointCloud.new([Point.new(0.0, 0.0, 0.0, color: {255, 0, 0})])
{:ok, ply} = ExCodecs.Spatial.encode(cloud, format: :ply)
{:ok, cloud} = ExCodecs.Spatial.decode(ply, format: :ply)
ExCodecs.Spatial.stream_decode(ply, format: :ply) |> Enum.to_list()
ExCodecs.Spatial.available_formats() #=> [:ply, :spatial_binary, :gsplat]
ExCodecs.codec_info(:ply) #=> {:ok, %ExCodecs.Codec{category: :spatial, interface: :spatial, ...}}

API Overview

Binary registry API

{:ok, encoded} = ExCodecs.encode(:zstd, binary, level: 3)
{:ok, decoded} = ExCodecs.decode(:zstd, compressed)

The binary registry API is always codec atom + binary.

Category APIs

• ExCodecs.Compression.compress/3 — compression terminology over the binary registry API.
• ExCodecs.Spatial.encode/2 / decode/2 — point clouds and Gaussians (struct↔format).

These are specialized entry points in the same framework, sharing conventions such as tagged results and %ExCodecs.Error{}.

encode/3 / decode/3

First argument is always a codec atom. Returns {:ok, binary} or {:error, %ExCodecs.Error{}}.

available_codecs/0

ExCodecs.available_codecs()          # all available catalog entries
ExCodecs.available_codecs(:spatial) #=> [:gsplat, :ply, :spatial_binary]

Returns a sorted list of codec atoms that are both registered and have a loaded implementation. Use available_codecs/1 to filter the shared catalog by category.

supports?/1

ExCodecs.supports?(:zstd)       #=> true
ExCodecs.supports?(:nonexistent) #=> false

Returns true only if the codec is registered and its native NIF is loaded.

codec_info/1

{:ok, info} = ExCodecs.codec_info(:zstd)
info.category       #=> :compression
info.native?        #=> true
info.streaming?     #=> false  # block API only today
info.configurable?  #=> true
info.version        #=> backend version string

Returns a structured %ExCodecs.Codec{} struct with metadata, or {:error, :unsupported_codec}.

Supported Codecs

Codec	Category	Configurable	Streaming	Options
:zstd	compression	Yes	No	level 1-22 (pure-Rust structured-zstd); max_output_size (default 256 MiB)
:lz4	compression	No	No	size-prepended lz4_flex; max_output_size
:snappy	compression	No	No	max_output_size
:bzip2	compression	Yes	No	block_size 1-9; max_output_size
:blosc2	compression	Yes	No	C-Blosc2 chunk only. cname / clevel / shuffle / typesize; max_output_size

Spatial formats

Format	Types	Notes
:ply	PointCloud / GaussianCloud	ASCII or binary PLY; Gaussian PLY properties
:spatial_binary	PointCloud	Compact little-endian EXCP container
:gsplat	GaussianCloud	Compact little-endian GSPL container

See Understanding Spatial Codecs and the frozen Spatial wire formats.

Known limitations

• Zstd uses pure-Rust structured-zstd. Levels 1–22 work, but compressed bytes/ratios are not guaranteed identical to C libzstd.
• Decompression defaults to a 256 MiB max_output_size. Raise it only for trusted inputs; do not decompress untrusted payloads without a tight limit.
• Spatial file stream_decode is incremental (bounded memory). In-memory binaries use chunked Rust unpack when available, otherwise materialize. stream_encode/2 still collects the enumerable, then encodes once; use encode_to_file/3 with an explicit :schema for EXCP/GSPL file streaming.

Architecture

ExCodecs is layered as follows:

1. Category APIs — ExCodecs provides the binary registry API; ExCodecs.Compression adds compression terminology; ExCodecs.Spatial handles spatial domain types and formats.

2. Codec Behaviour (ExCodecs.Codec) — encode/2 / decode/2 on binaries; optional __codec_info__/0 for registry metadata.

3. Shared codec catalog (ExCodecs.CodecRegistry) — ETS map of codec atoms to modules, categories, interface shapes, and metadata, populated at startup.

4. Native NIFs (ExCodecs.Native) — pure-Rust compression plus optional spatial DirtyCpu / mmap acceleration via rustler_precompiled (or local compile).

5. Category discovery — available_codecs/0 lists the whole catalog; available_codecs/1 filters it, and ExCodecs.Spatial.available_formats/0 provides the spatial category's preferred built-in order.

To add a binary codec: implement ExCodecs.Codec, wire a NIF if needed, and register it with interface: :binary. Spatial implementations live under ExCodecs.Spatial and register with category: :spatial and interface: :spatial.

Error Handling

All public functions return {:ok, result} or {:error, %ExCodecs.Error{}}. Error reasons are atoms:

Reason	Meaning
:unsupported_codec	The codec name is not registered
:codec_unavailable	Registered but the native NIF failed to load
:invalid_data	Data is not a binary or is otherwise invalid
:invalid_options	Options are out of range or malformed
:compression_failed	The underlying compression operation failed
:decompression_failed	The underlying decompression operation failed
:nif_not_loaded	The NIF library could not be loaded
:io_error	File read/write failure
:truncated_input	Incomplete binary
:output_limit_exceeded	Decompress would exceed max_output_size

{:error, error} = ExCodecs.encode(:unknown, "data")
error.reason  #=> :unsupported_codec

{:error, error} = ExCodecs.encode(:zstd, "data", level: 99)
error.reason  #=> :invalid_options

Benchmarking

ExCodecs includes benchmarking utilities via benchee.

# Run all compression benchmarks
mix benchmarks

Benchmarks are defined in bench/ and run in the :bench environment. Results are saved to bench/results/ (git-ignored).

Development

# Fetch dependencies and compile (includes NIF compilation)
mix deps.get && mix compile

# Run the full test suite
mix test

# Run tests with coverage
mix coveralls

# Run static analysis
mix credo
mix dialyzer

# Format code
mix format

# Lint Rust NIF code
mix rust.lint

# Run Rust NIF tests
mix rust.test

# Generate documentation
mix docs

Requirements

• Elixir 1.17+
• Erlang/OTP 26+
• Rust 1.94+ (only required if precompiled NIFs are unavailable for your platform)

License

Apache License, Version 2.0. See LICENSE for details.