Add Zarr v3 support

.coderabbit.yaml

@@ -0,0 +1,3 @@
+reviews:
+  path_filters:
+    - "!test/data/**/*.zarr/**"

.github/workflows/MainDistributionPipeline.yml

@@ -21,6 +21,7 @@ jobs:
       duckdb_version: v1.5.0
       ci_tools_version: v1.5.0
       extension_name: duckdb_zarr
+      opt_in_archs: linux_amd64;linux_arm64;osx_amd64;osx_arm64;windows_amd64;windows_amd64_mingw;wasm_mvp;wasm_eh;wasm_threads
 
   code-quality-check:
     name: Code Quality Check

.github/workflows/PublishExtensionRepository.yml

@@ -24,6 +24,7 @@ jobs:
       duckdb_version: v1.5.0
       ci_tools_version: v1.5.0
       extension_name: duckdb_zarr
+      opt_in_archs: linux_amd64;linux_arm64;osx_amd64;osx_arm64;windows_amd64
 
   package-repository:
     name: Package release assets

.gitignore

@@ -7,3 +7,6 @@ testext
 test/python/__pycache__/
 scripts/__pycache__/
 .Rhistory
+__pycache__/
+duckdb_zarr-v1.5.0-*.duckdb_extension.gz
+duckdb_zarr.duckdb_extension

ARCHITECTURE.md

@@ -28,9 +28,9 @@
 
 ## Current MVP
 
-- Implemented now: local Zarr v2 metadata discovery, consolidated `.zmetadata` support, gzip/uncompressed numeric chunk decode, and DuckDB table functions (`zarr_groups`, `zarr_arrays`, `zarr_chunks`, `zarr_cells`) with projection-aware scans and chunk pruning.
-- Partially implemented: the Store Adapter includes practical remote support for consolidated HTTP/S3-style stores via DuckDB `httpfs`, and the Planner path performs predicate pushdown for the current `zarr_cells()` scan.
-- Planned: the Arrow Materializer, broader codec chains such as Blosc, richer convention adapters, and the fuller target flow described below.
+- Implemented now: local Zarr v2 and v3 metadata discovery, Zarr v2 consolidated `.zmetadata` support, gzip/uncompressed numeric chunk decode, and DuckDB table functions (`zarr_groups`, `zarr_arrays`, `zarr_chunks`, `zarr_cells`, `zarr`) with projection-aware scans and chunk pruning.
+- Partially implemented: the Store Adapter includes practical remote support for consolidated HTTP/S3-style Zarr v2 stores via DuckDB `httpfs`, and the Codec Pipeline now handles the current v3 subset of `bytes`, optional `gzip`, constrained `transpose`, and `sharding_indexed` with inner Blosc (`zstd`) for common OME-Zarr image and label data.
+- Planned: the Arrow Materializer, broader codec chains beyond the current Blosc path, richer convention adapters, and the fuller target flow described below.
 
 ## Flow
 

Makefile

@@ -5,13 +5,16 @@ PROJ_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
 EXT_NAME=duckdb_zarr
 EXT_CONFIG=${PROJ_DIR}extension_config.cmake
 
-.PHONY: fixture test_metadata
+.PHONY: fixture test_metadata test_python_example
 
 fixture:
 	python3 scripts/create_sample_zarr.py
 
 test_metadata: fixture
 	${MAKE} test
 
+test_python_example: fixture
+	uvx --with duckdb==1.5.0 --with numpy --with pandas python test/python_smoke.py
+
 # Include the Makefile from extension-ci-tools
 include extension-ci-tools/makefiles/duckdb_extension.Makefile

README.md

@@ -11,11 +11,11 @@ The current implementation follows the project documents conservatively:
 Today’s extension provides metadata and relational scan table functions for local stores and a constrained remote read path:
 
 - `zarr(path)` for a simple array overview
-- `zarr(path, array_path)` as a convenience alias for `zarr_cells(path, array_path)`
-- `zarr_groups(path)`
-- `zarr_arrays(path)`
-- `zarr_chunks(path)`
-- `zarr_cells(path, array_path)` for Zarr dtypes `|b1`, `i1`, `i2`, `i4`, `i8`, `u1`, `u2`, `u4`, `u8`, `f2`, `f4`, and `f8`
+- `zarr(path, array_path)` and `zarr(path, array_path, version_override)` as convenience aliases for `zarr_cells(...)`
+- `zarr_groups(path)` and `zarr_groups(path, version_override)`
+- `zarr_arrays(path)` and `zarr_arrays(path, version_override)`
+- `zarr_chunks(path)` and `zarr_chunks(path, version_override)`
+- `zarr_cells(path, array_path)` and `zarr_cells(path, array_path, version_override)` for Zarr dtypes `|b1`, `i1`, `i2`, `i4`, `i8`, `u1`, `u2`, `u4`, `u8`, `f2`, `f4`, and `f8`
 
 This gives a usable SQL entrypoint for understanding a Zarr store and projecting dense numeric arrays into relational rows. Other dtypes such as complex values and other unsupported or non-standard Zarr dtypes are currently rejected by `zarr_cells(path, array_path)`.
 
@@ -24,14 +24,21 @@ This gives a usable SQL entrypoint for understanding a Zarr store and projecting
 The MVP currently supports:
 
 - Local filesystem Zarr v2 discovery
+- Local filesystem Zarr v3 discovery from `zarr.json`
 - Zarr v2 consolidated metadata discovery from `.zmetadata`
 - Remote `http://`, `https://`, and `s3://` stores when DuckDB `httpfs` is available and the store is consolidated
 - OME-Zarr-style multiscales groups built on Zarr v2, including level arrays such as `0`
+- Real-world OME-Zarr v3 metadata discovery for stores such as image-label hierarchies and multiscales groups
 - Group enumeration from `.zgroup`
-- Array enumeration from `.zarray`
-- Chunk enumeration for both `.` and `/` dimension separators
+- Array enumeration from `.zarray` and `zarr.json`
+- Automatic v2/v3 detection for local stores, with explicit `version_override` support (`auto`, `v2`, `v3`) on the lower-level SQL functions
+- Chunk enumeration for v2 `.` and `/` separators plus Zarr v3 `default` and `v2` chunk-key encodings
 - `zarr_cells(path, array_path)` for dense arrays with dtypes `|b1`, `i1`, `i2`, `i4`, `i8`, `u1`, `u2`, `u4`, `u8`, `f2`, `f4`, and `f8`
 - Uncompressed and gzip-compressed chunk payloads
+- Zarr v3 regular chunk grids with `bytes` and optional `gzip` codecs
+- Zarr v3 transpose handling for identity and full-reverse permutations
+- Zarr v3 `sharding_indexed` arrays with inner Blosc (`zstd`) decode for common OME-Zarr image and label data
+- Real OME-Zarr v3 cell scans against `test/data/idr0062A/6001240_labels.zarr`
 - Missing-chunk fill-value materialization for supported `zarr_cells()` dtypes when `fill_value` is present
 - Dynamic `(dim_0, ..., value)` projection based on array rank and dtype
 - Projection-aware `zarr_cells()` scans
@@ -43,9 +50,9 @@ The MVP currently supports:
 The MVP does not yet support:
 
 - `zarr_cells()` for dtypes outside `|b1`, `i1`, `i2`, `i4`, `i8`, `u1`, `u2`, `u4`, `u8`, `f2`, `f4`, and `f8`
-- Blosc chunk decode
+- Blosc codecs outside the currently supported `zstd`-backed path
 - Non-consolidated remote store discovery
-- Zarr v3 metadata
+- Remote hierarchical discovery for non-consolidated Zarr v3 group stores
 - Arrow materialization
 
 ## Quick Start
@@ -60,10 +67,10 @@ Install developer tooling for formatting and hooks:
 
 ```sh
 python3 -m pip install pre-commit clang-format==11.0.1
-pre-commit install
+pre-commit install --hook-type pre-commit --hook-type pre-push
 ```
 
-That single `pre-commit install` command now installs both:
+Re-run that command after pulling hook configuration changes. It installs both:
 - a `pre-commit` hook that auto-fixes DuckDB formatting
 - a `pre-push` hook that runs the same formatter in `--check` mode before CI sees the push
 
@@ -79,6 +86,18 @@ Run the extension tests:
 make test_metadata
 ```
 
+Run the Python smoke example with the same environment shape many Python users will use:
+
+```sh
+uvx --with duckdb==1.5.0 --with numpy --with pandas python test/python_smoke.py
+```
+
+There is also a matching make target:
+
+```sh
+make test_python_example
+```
+
 Package a static extension repository layout from downloaded CI artifacts:
 
 ```sh
@@ -115,7 +134,14 @@ SELECT * FROM zarr_arrays('test/data/simple_v2.zarr');
 SELECT * FROM zarr_chunks('test/data/simple_v2.zarr');
 SELECT * FROM zarr_cells('test/data/simple_v2.zarr', 'temperature');
 SELECT * FROM zarr('test/data/simple_v2.zarr');
+SELECT * FROM zarr_arrays('test/data/simple_v3.zarr', 'v3');
+SELECT * FROM zarr('test/data/simple_v3.zarr', 'temperature_v3', 'v3');
 SELECT * FROM zarr('test/data/ome_example.ome.zarr', '0');
+SELECT * FROM zarr('test/data/idr0062A/6001240_labels.zarr');
+SELECT * FROM zarr('test/data/idr0062A/6001240_labels.zarr', '0') LIMIT 5;
+SELECT SUM(value)
+FROM zarr('test/data/idr0062A/6001240_labels.zarr', 'labels/0/0')
+WHERE dim_0 = 0 AND dim_1 = 0 AND dim_2 < 4 AND dim_3 < 4;
 ```
 
 For remote stores in this phase, the practical contract is:
@@ -124,6 +150,13 @@ For remote stores in this phase, the practical contract is:
 - DuckDB must have `httpfs` available for the URI scheme you use
 - chunk data are still read directly from remote chunk object paths
 
+Version detection notes:
+
+- `zarr(path)` auto-detects between local v2 and v3 stores
+- lower-level functions accept an optional `version_override` argument: `auto`, `v2`, or `v3`
+- the convenience cell entrypoint also accepts an override as `zarr(path, array_path, version_override)`
+- explicit override is useful when a path is ambiguous or when you want failures to be version-specific
+
 ## Install Like An Extension
 
 This repo now includes a release-asset publishing workflow in
@@ -209,7 +242,7 @@ The current code maps directly to the architecture document:
 
 - Store Adapter: local filesystem traversal through DuckDB’s `FileSystem`
 - Store Adapter: consolidated remote discovery for `http://`, `https://`, and `s3://` paths through DuckDB filesystem routing
-- Metadata Parser: `.zgroup` and `.zarray` parsing via `yyjson`
+- Metadata Parser: Zarr v2 `.zgroup`/`.zarray` plus Zarr v3 `zarr.json` parsing via `yyjson`
 - DuckDB Bridge: table functions registered from the extension entrypoint
 - Relational Cell Scan: `zarr_cells()` with pushed filters, projection-aware output, and dimension-based chunk pruning
 

ROADMAP.md

@@ -4,12 +4,17 @@
 - [x] zarr_groups
 - [x] zarr_arrays
 - [x] local store discovery
+- [x] local Zarr v3 metadata discovery
 
 ## [x] Phase 2: Cell Scan
 - [x] zarr_cells
 - [x] numeric types
-- [~] gzip
-- [ ] blosc
+- [x] gzip
+- [x] automatic v2/v3 detection
+- [x] explicit version override support
+- [x] constrained Zarr v3 codec support (`bytes`, optional `gzip`, constrained `transpose`)
+- [x] Zarr v3 `sharding_indexed` with Blosc (`zstd`) for local OME-Zarr-style arrays
+- [~] broader blosc coverage
 
 ## [x] Phase 3: Optimization
 - [x] chunk pruning
@@ -18,6 +23,7 @@
 
 ## [~] Phase 4: Remote
 - [~] S3 / HTTP for consolidated stores
+- [ ] non-consolidated Zarr v3 remote discovery
 - [ ] caching
 
 ## [ ] Phase 5: Advanced

scripts/create_sample_zarr.py

@@ -11,6 +11,7 @@
 
 ROOT = Path(__file__).resolve().parents[1]
 STORE = ROOT / "test" / "data" / "simple_v2.zarr"
+STORE_V3 = ROOT / "test" / "data" / "simple_v3.zarr"
 OME_STORE = ROOT / "test" / "data" / "ome_example.ome.zarr"
 
 
@@ -198,8 +199,63 @@ def create_ome_store() -> None:
     )
 
 
+def create_simple_v3_store() -> None:
+    shutil.rmtree(STORE_V3, ignore_errors=True)
+    root_group = {"zarr_format": 3, "node_type": "group", "attributes": {"name": "simple_v3"}}
+    write_json(STORE_V3 / "zarr.json", root_group)
+
+    temperature_array = {
+        "zarr_format": 3,
+        "node_type": "array",
+        "shape": [3, 2],
+        "data_type": "float64",
+        "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": [2, 2]}},
+        "chunk_key_encoding": {"name": "default", "configuration": {"separator": "/"}},
+        "fill_value": None,
+        "codecs": [
+            {"name": "bytes", "configuration": {"endian": "little"}},
+            {"name": "gzip", "configuration": {"level": 1}},
+        ],
+        "attributes": {"units": "celsius"},
+    }
+    write_json(STORE_V3 / "temperature_v3" / "zarr.json", temperature_array)
+    write_chunk(STORE_V3 / "temperature_v3" / "c" / "0" / "0", gzip_chunk(struct.pack("<4d", 1.0, 2.0, 3.0, 4.0)))
+    write_chunk(STORE_V3 / "temperature_v3" / "c" / "1" / "0", gzip_chunk(struct.pack("<4d", 5.0, 6.0, 0.0, 0.0)))
+
+    mask_array = {
+        "zarr_format": 3,
+        "node_type": "array",
+        "shape": [2, 3],
+        "data_type": "bool",
+        "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": [2, 2]}},
+        "chunk_key_encoding": {"name": "v2", "configuration": {"separator": "."}},
+        "fill_value": False,
+        "codecs": [{"name": "bytes"}],
+    }
+    write_json(STORE_V3 / "mask_v3" / "zarr.json", mask_array)
+    write_chunk(STORE_V3 / "mask_v3" / "0.0", bytes([1, 0, 1, 1]))
+    write_chunk(STORE_V3 / "mask_v3" / "0.1", bytes([0, 0, 1, 0]))
+
+    fortran_array = {
+        "zarr_format": 3,
+        "node_type": "array",
+        "shape": [2, 2],
+        "data_type": "int32",
+        "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": [2, 2]}},
+        "chunk_key_encoding": {"name": "default", "configuration": {"separator": "/"}},
+        "fill_value": 0,
+        "codecs": [
+            {"name": "transpose", "configuration": {"order": [1, 0]}},
+            {"name": "bytes", "configuration": {"endian": "little"}},
+        ],
+    }
+    write_json(STORE_V3 / "fortran_v3" / "zarr.json", fortran_array)
+    write_chunk(STORE_V3 / "fortran_v3" / "c" / "0" / "0", struct.pack("<4i", 1, 3, 2, 4))
+
+
 def main() -> None:
     create_simple_store()
+    create_simple_v3_store()
     create_ome_store()
 
 

src/include/zarr_metadata.hpp

@@ -1,17 +1,18 @@
 #pragma once
 
+#include "duckdb/function/function_set.hpp"
 #include "duckdb/function/table_function.hpp"
 
 namespace duckdb {
 
 class ZarrMetadata {
 public:
-	static TableFunction GetGroupsFunction();
-	static TableFunction GetArraysFunction();
-	static TableFunction GetChunksFunction();
-	static TableFunction GetCellsFunction();
-	static TableFunction GetZarrFunction();
-	static TableFunction GetZarrCellsAliasFunction();
+	static TableFunctionSet GetGroupsFunction();
+	static TableFunctionSet GetArraysFunction();
+	static TableFunctionSet GetChunksFunction();
+	static TableFunctionSet GetCellsFunction();
+	static TableFunctionSet GetZarrFunction();
+	static TableFunctionSet GetZarrCellsAliasFunction();
 };
 
 } // namespace duckdb