Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 50 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,54 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [0.2.3] - 2026-07-18

### Added

- **Incremental spatial file I/O** — EXCP (`:spatial_binary`), GSPL (`:gsplat`),
and PLY `stream_decode` with `source: :file` (or `:auto` path detection) read
the header then **one record/vertex at a time** from disk (bounded memory).
Binary PLY uses a fixed stride; ASCII PLY reads lines.
- `Binary.stream_encode_to_file/3` and `Gsplat.stream_encode_to_file/3` —
incremental file writes with an explicit `:schema` (placeholder header,
seek-back count). `Spatial.Stream.encode_to_file/3` uses these when
`:schema` is present with `format: :spatial_binary` or `:gsplat`.
- **Spatial Rust acceleration** (DirtyCpu NIFs): chunked EXCP/GSPL pack &
unpack, mmap-backed file `stream_decode`, binary PLY body unpack, and
chunked `stream_encode_to_file`. Pass `accel: false` to force pure Elixir.
Property tests compare both backends byte-for-byte / structurally.

### Changed

- Mix `preferred_cli_env` moved into `cli/0` (`preferred_envs`) for Mix 1.20+.
- In-memory spatial `stream_decode` uses chunked Rust unpack when the spatial
NIF is loaded; otherwise it still materializes through `decode/2`.

### Notes

- Wire layouts for EXCP / GSPL / PLY remain the **v0.2.0 freeze** in
`docs/spatial_formats.md` (Rust output is byte-compatible).
- Precompiled NIF checksums must be regenerated when publishing GitHub release
artifacts for `0.2.3`.

## [0.2.2] - 2026-07-17

### Notes

- Version number reserved and **superseded; not published** to Hex.pm. The
work intended for 0.2.2 was rolled into 0.2.3 (spatial streaming and Rust
acceleration) instead. Recorded per [Keep a Changelog](https://keepachangelog.com)
so the version-skip is not silent.

## [0.2.1] - 2026-07-17

### Fixed

- README links and badge URLs corrected following the 0.2.0 spatial release.
- `mix.exs` version bump to 0.2.1.

## [0.2.0] - 2026-07-16

### Added
Expand Down Expand Up @@ -96,7 +144,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- README badges updated with CI, Hex.pm, docs, license, Elixir version, and Coveralls
coverage links.

## [0.1.0] - 2025-06-09
## [0.1.0] - 2026-06-13

### Added

Expand All @@ -107,7 +155,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Blosc2 shuffle support (`:none`, `:byte`).
- Rust NIF implementation via `rustler_precompiled`.
- Precompiled binaries for macOS (ARM64, x86_64), Linux (glibc, musl, ARM64), Windows (x86_64).
- `ExCodecs.Compression` convenience module (`compress/2`, `decompress/2`).
- `ExCodecs.Compression` convenience module (`compress/3`, `decompress/3`).
- Structured error handling with `%ExCodecs.Error{}`.
- 154 tests (unit + property-based with StreamData).
- 90%+ test coverage.
Expand Down
12 changes: 8 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Add `ex_codecs` to your list of dependencies in `mix.exs`:
```elixir
def deps do
[
{:ex_codecs, "~> 0.2.0"}
{:ex_codecs, "~> 0.2.3"}
]
end
```
Expand Down Expand Up @@ -180,7 +180,10 @@ and the frozen [Spatial wire formats](https://hexdocs.pm/ex_codecs/spatial_forma
bytes/ratios are not guaranteed identical to C libzstd.
- **Decompression** defaults to a **256 MiB** `max_output_size`. Raise it only
for trusted inputs; do not decompress untrusted payloads without a tight limit.
- Spatial `stream_*` helpers **materialize** full payloads today.
- Spatial **file** `stream_decode` is incremental (bounded memory). In-memory
binaries use chunked Rust unpack when available, otherwise materialize.
`stream_encode/2` still collects the enumerable, then encodes once; use
`encode_to_file/3` with an explicit `:schema` for EXCP/GSPL file streaming.

## Architecture

Expand All @@ -196,8 +199,9 @@ ExCodecs is layered as follows:
3. **Shared codec catalog** (`ExCodecs.CodecRegistry`) — ETS map of codec atoms
to modules, categories, interface shapes, and metadata, populated at startup.

4. **Native NIFs** (`ExCodecs.Native`) — pure-Rust compression via
`rustler_precompiled` (or local compile).
4. **Native NIFs** (`ExCodecs.Native`) — pure-Rust compression plus optional
spatial DirtyCpu / mmap acceleration via `rustler_precompiled` (or local
compile).

5. **Category discovery** — `available_codecs/0` lists the whole catalog;
`available_codecs/1` filters it, and
Expand Down
43 changes: 23 additions & 20 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,8 @@ The implementation is layered in four tiers:
modules expose the category's struct↔format contract.

4. **Native layer** -- Compression modules delegate to the Rustler NIF
(`ExCodecs.Native`). Spatial codecs are Elixir implementations in v0.2.0.
(`ExCodecs.Native`). Spatial codecs keep a pure-Elixir path and, since
v0.2.3, optional DirtyCpu / mmap acceleration via the same NIF crate.

---

Expand Down Expand Up @@ -576,25 +577,25 @@ ExCodecs.Error.error(:invalid_data, message: "Data must be a binary")
### NIF error mapping

The Rust side returns errors as atoms: `{:error, :compression_failed}`,
`{:error, :invalid_data}`, etc. The Elixir codec modules or the framework
map these into structured errors via `ExCodecs.Error.from_nif/2`:
`{:error, :invalid_data}`, `{:error, :output_limit_exceeded}`, etc. Elixir
codec modules call `ExCodecs.NIF.wrap/2` (or `safe_call/2`) to turn those into
`{:error, %ExCodecs.Error{}}`:

```elixir
def from_nif({:error, reason}, codec) when is_atom(codec) do
{:error, %__MODULE__{
reason: nif_error_to_atom(reason),
message: "NIF error in codec #{codec}: #{inspect(reason)}",
codec: codec,
details: reason
}}
end
# Typical codec decode path
case ExCodecs.NIF.max_output_size(opts) do
{:ok, max} ->
ExCodecs.NIF.wrap(:zstd, ExCodecs.Native.zstd_decompress(data, max))

defp nif_error_to_atom(reason) when is_atom(reason), do: reason
defp nif_error_to_atom(_), do: :compression_failed
{:error, _} = err ->
err
end
```

This creates a boundary: the Rust layer communicates errors as atoms, and the
Elixir layer enriches those atoms into structured errors with context.
`NIF.wrap/2` maps known atoms to structured errors with a default message and
`codec:` field. Unknown atoms still become `%ExCodecs.Error{}` with the raw
atom preserved. This keeps a clear boundary: Rust communicates status as
atoms; Elixir enriches them for callers.

### Error flow

Expand Down Expand Up @@ -727,12 +728,14 @@ predictable and navigable.

## Codec Categories

### Spatial (implemented in 0.2.0)
### Spatial (since 0.2.0; Rust accel in 0.2.3)

Spatial codecs map structured geometric types to interchange formats. They are
pure Elixir and use the specialized `ExCodecs.Spatial` API rather than forcing
structs through the binary-only `ExCodecs.Codec` callbacks. They are registered
in the shared catalog with `category: :spatial` and `interface: :spatial`.
Spatial codecs map structured geometric types to interchange formats. They use
the specialized `ExCodecs.Spatial` API rather than forcing structs through the
binary-only `ExCodecs.Codec` callbacks. They are registered in the shared
catalog with `category: :spatial` and `interface: :spatial`. Hot paths may use
DirtyCpu pack/unpack and mmap-backed file streams when the NIF is loaded
(`accel: false` forces Elixir).

`ExCodecs.available_codecs/0` lists all available entries,
`ExCodecs.available_codecs(:spatial)` filters the shared catalog, and
Expand Down
Loading
Loading