diff --git a/packages/zarr-metadata/README.md b/packages/zarr-metadata/README.md
index a01ed26842..4b3c1293fe 100644
--- a/packages/zarr-metadata/README.md
+++ b/packages/zarr-metadata/README.md
@@ -2,410 +2,44 @@
Documentation:
-Basic tools for modelling Zarr metadata, with minimal dependencies.
+Types, models and validators for Zarr metadata documents, with minimal
+dependencies.
`zarr-metadata` is developed in the
[zarr-python repository](https://github.com/zarr-developers/zarr-python/tree/main/packages/zarr-metadata)
-and released independently of `zarr` itself. Install it with:
+and released independently of `zarr`. Install it with:
```
pip install zarr-metadata
```
-## Who needs this
-
-This library supports software that interacts with Zarr metadata documents.
-
## What this is
-This is *not* a full Zarr implementation. It's a collection of data structures and routines that
-closely model the content of the Zarr specifications, such as:
-
-- **Typed JSON shapes** ([`zarr_metadata.v2`](https://zarr-metadata.readthedocs.io/en/latest/api/v2/) and
- [`zarr_metadata.v3`](https://zarr-metadata.readthedocs.io/en/latest/api/v3/)): `TypedDict` definitions and
- `Literal` aliases for the JSON documents specified by the
- [Zarr v2](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html) and
- [Zarr v3](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html)
- specifications, plus types for
- [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/) and a
- few widely-used-but-unspecified entities (e.g. consolidated metadata).
-- **Document models** ([`zarr_metadata.model`](https://zarr-metadata.readthedocs.io/en/latest/api/model/)): immutable
- models of whole metadata documents, each one a document and the scope
- it was read in, with readers that report every problem at its
- location, and store-key (de)serialization. A document
- produced by `to_json` shares no mutable state with the model that produced
- it.
-- **Definitions and scopes** ([`zarr_metadata.definition`](https://zarr-metadata.readthedocs.io/en/latest/api/definition/),
- with each format's kinds in `zarr_metadata.v3.definition` and
- `zarr_metadata.v2.definition`): each extension's configuration as a
- `TypedDict` with its rules, and the scope that decides which definitions
- read a document. `zarr_metadata.v3` and `zarr_metadata.v2` export every
- definition and scope of their format, such as `BYTES_CODEC` and
- `CORE_AND_EXTENSIONS_CONTEXT`.
-- **JSON type checking** ([`zarr_metadata.typed_json`](https://zarr-metadata.readthedocs.io/en/latest/api/typed_json/)):
- `check` checks a JSON value against any `TypedDict` and reports every
- problem with its location.
-- **Problems** ([`zarr_metadata.problems`](https://zarr-metadata.readthedocs.io/en/latest/api/problems/)): the problem
- values every reader returns, and the error that carries them.
-- **JSON Schema** ([`zarr_metadata.json_schema`](https://zarr-metadata.readthedocs.io/en/latest/api/json_schema/)): the
- JSON Schema of a `TypedDict`, a v3 declaration, or a whole `zarr.json`.
-- **Optional Pydantic integration** ([`zarr_metadata.pydantic`](https://zarr-metadata.readthedocs.io/en/latest/api/pydantic/),
- requires Pydantic 2.13 or newer): each model as a Pydantic field type that
- validates raw documents through the same strict parser.
-
-## What this is for
-
-- **Static Type Checking** The public `TypedDict` definitions describe the static JSON shape of Zarr
- metadata.
-
-- **Dynamic JSON validation** For strict, loc-aware validation of JSON loaded from disk, use the
- model parser:
-
- ```python test="true" session="readme"
- import json
- from zarr_metadata.model import ZarrV3ArrayMetadata
-
- with open("zarr.json", "rb") as f:
- raw = json.load(f)
-
- metadata = ZarrV3ArrayMetadata.from_json(raw)
- ```
-
- The optional Pydantic integration delegates raw input to the same strict
- parser and returns the same normalized model class:
-
- ```python test="true" session="readme"
- from pydantic import TypeAdapter
- import zarr_metadata.pydantic as zmp
-
- adapter: TypeAdapter[zmp.ZarrV3ArrayMetadataField] = TypeAdapter(zmp.ZarrV3ArrayMetadataField)
- metadata = adapter.validate_python(raw)
- encoded = metadata.to_key_value()["zarr.json"]
- ```
-
- A field annotated `Annotated[ZarrV3ArrayMetadata, zmp.InScope(CONTEXT)]`
- is read in `CONTEXT`, whatever the validation context holds, and a model
- instance given to it is read again in `CONTEXT`.
-
- Note that a bare `TypeAdapter` over a public `TypedDict` is a **coercive** shape
- adapter, not a Zarr conformance validator; it may coerce values or discard
- members that the strict model parser rejects. This is better suited for handling input provided
- from Python objects (e.g., user-authored dicts) than JSON.
-
-## Validation boundary
-
-The readers and model constructors enforce the declared document structure and a small set
-of context-free consistency rules, e.g.:
-- fixed format literals,
-- finite JSON numbers outside `attributes` and the members nothing in scope reads
-- non-negative dimensions
-- one `dimension_names` entry per array dimension.
-
-In a v3 document, validation also checks each extension point: the data
-type, the chunk grid, the chunk key encoding, each codec, and each storage
-transformer. Each one is checked by the definition that claims its name in
-a scope. The scope is `CORE_AND_EXTENSIONS_CONTEXT` unless a `context` is passed.
-`ZARR_PYTHON_CONTEXT` adds the data types that zarr-python writes but no
-specification registers.
-If the definition rejects a configuration, the reader rejects it too,
-and a configuration key the definition does not declare is reported as
-`unknown_key`. If no definition in the scope claims a name, the reader
-does not check that extension point, and the consumer decides whether to
-support it.
-
-Some checks relate one part of a v3 document to another:
-
-- The fill value is checked against the data type the document names, by
- that data type's definition.
-- The chunk grid is checked against the shape, by the grid's definition.
-- The codecs are checked as a pipeline. In order, each codec's definition
- checks the codec against the chunk it receives. This includes a shard's
- inner codecs and index codecs.
-
-The readers do no arithmetic on values. For example, they do not check
-whether a fill value survives a `cast_value` round trip.
-
-Some choices that the spec text leaves open, or answers in two ways:
-
-- **`attributes` may hold `NaN`, `Infinity` and `-Infinity`.** The spec
- does not interpret attributes, and zarr-python and xarray write these
- numbers there (for example, a CF `_FillValue`). The models read them,
- and `to_key_value` writes them back as bare tokens, which a strict JSON
- parser rejects. So may a member the spec does not define when no
- definition in scope claims it: nothing reads it, so it is kept as
- written. `check`, from `zarr_metadata.typed_json`, rejects a non-finite
- number anywhere, including in attributes.
-- **A reader walks at most 256 levels of nesting.** A value nested deeper
- is an `invalid_value` problem at the first level past the limit,
- wherever it is. Every reader, writer and comparison uses one stack
- frame per level, and `copy.deepcopy` (and `pickle` before Python 3.12)
- uses two. A document at the limit therefore uses about half of the
- interpreter's default recursion limit, and the rest is left for the
- caller.
-- **`consolidated_metadata: null` is a problem.** A zarr-python 3.0.x bug
- wrote it. The spec requires an object, and the package accepts only an
- object. A reader of those stores removes the key before reading.
-- **An extension name must follow the spec's naming rule**,
- `^[a-z][a-z0-9-_.]+$`, or be a URI, which earlier versions of the spec
- required. Any other name is rejected before any definition is
- consulted, so `""` and `"foo/bar"` are problems, not unknown extensions.
-- **`must_understand: false` is rejected at every extension point**,
- including codecs and storage transformers, although the core spec names
- only the data type, chunk grid and chunk key encoding. A reader that
- skips a codec reads wrong bytes, just as a reader that skips a data type
- reads wrong values. On an unknown top-level member, `must_understand`
- keeps its meaning: a reader can skip that member.
-- **A v2 datetime or timedelta dtype must have a unit**, as in ` Iterator[ValidationProblem]:
+ if "dictionary" in configuration and "dictionary_size" not in configuration:
+ yield ValidationProblem(
+ ("dictionary_size",), "a dictionary needs its size", "missing_key"
+ )
+
+
+ACME_LZ4 = CodecDefinition(
+ name="acme.lz4",
+ configuration=AcmeLz4Configuration,
+ kind="bytes_bytes",
+ size="dynamic",
+ rules=acme_lz4_rules,
+)
+CONTEXT = CORE_AND_EXTENSIONS_CONTEXT.update(codecs=[*CORE_AND_EXTENSIONS_CONTEXT.codecs.values(), ACME_LZ4])
+```
+
+Pass `context=CONTEXT` to any reader or model constructor, and a document
+that names `acme.lz4` is checked against this definition.
+
+Some rules for the pieces:
+
+- The `TypedDict` must be a `typing_extensions.TypedDict`, and must say
+ how undeclared keys are handled: `closed=True`, `extra_items=`, or
+ `closed=False`. An open-by-default `TypedDict` is refused, because it
+ would accept a misspelled key silently.
+- Its members must be JSON shapes: `int`, `float`, `bool`, `str`, `None`,
+ `JSONValue`, a `Literal`, `tuple[T, ...]`, `tuple[T1, T2]`, a union, a
+ `TypedDict`, `Mapping[str, V]`, a `NewType` or a type alias. A number
+ may carry bounds from annotated-types (`Gt`, `Ge`, `Lt`, `Le`,
+ `Interval`). Any other `Annotated` metadata is refused when the
+ definition is built.
+- The rules run only on a configuration within its bounds, so a rule can
+ rely on them. A rule reports the location of a problem, and the value
+ there becomes the problem's `input`. Show values in messages with
+ `zarr_metadata.problems.shown`, so they read like the package's own.
+- Define each function at module top level. A model holds the definitions
+ that read it, so it can be pickled only if they can.
+
+A data type also declares its fill value (`fill_value`,
+`fill_value_rules`, `fill_value_canonical`) and how its values are stored
+(`storage`). A chunk grid declares which shapes it fits (`shape_rules`) and
+its chunk lengths (`chunk_lengths`). A codec declares what it checks about
+the chunk it receives (`chunk_rules`), what it hands on (`transition`), and
+any pipelines it contains (`pipelines`). The
+[`zarr_metadata.v3.definition`](api/v3/definition.md) docstring describes
+each of these.
diff --git a/packages/zarr-metadata/docs/index.md b/packages/zarr-metadata/docs/index.md
index b139dfd307..50058a3041 100644
--- a/packages/zarr-metadata/docs/index.md
+++ b/packages/zarr-metadata/docs/index.md
@@ -1,411 +1,89 @@
# zarr-metadata
-Basic tools for modelling Zarr metadata, with minimal dependencies.
+Types, models and validators for Zarr metadata documents, with minimal
+dependencies.
`zarr-metadata` is developed in the
[zarr-python repository](https://github.com/zarr-developers/zarr-python/tree/main/packages/zarr-metadata)
-and released independently of `zarr` itself. Install it with:
+and released independently of `zarr`. Install it with:
```
pip install zarr-metadata
```
-## Who needs this
-
-This library supports software that interacts with Zarr metadata documents.
-
## What this is
-This is *not* a full Zarr implementation. It's a collection of data structures and routines that
-closely model the content of the Zarr specifications, such as:
-
-- **Typed JSON shapes** ([`zarr_metadata.v2`](api/v2.md) and
- [`zarr_metadata.v3`](api/v3/index.md)): `TypedDict` definitions and
- `Literal` aliases for the JSON documents specified by the
- [Zarr v2](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html) and
- [Zarr v3](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html)
- specifications, plus types for
- [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/) and a
- few widely-used-but-unspecified entities (e.g. consolidated metadata).
-- **Document models** ([`zarr_metadata.model`](api/model.md)): immutable
- models of whole metadata documents, each one a document and the scope
- it was read in, with readers that report every problem at its
- location, and store-key (de)serialization. A document
- produced by `to_json` shares no mutable state with the model that produced
- it.
-- **Definitions and scopes** ([`zarr_metadata.definition`](api/definition.md),
- with each format's kinds in `zarr_metadata.v3.definition` and
- `zarr_metadata.v2.definition`): each extension's configuration as a
- `TypedDict` with its rules, and the scope that decides which definitions
- read a document. `zarr_metadata.v3` and `zarr_metadata.v2` export every
- definition and scope of their format, such as `BYTES_CODEC` and
- `CORE_AND_EXTENSIONS_CONTEXT`.
-- **JSON type checking** ([`zarr_metadata.typed_json`](api/typed_json.md)):
- `check` checks a JSON value against any `TypedDict` and reports every
- problem with its location.
-- **Problems** ([`zarr_metadata.problems`](api/problems.md)): the problem
- values every reader returns, and the error that carries them.
-- **JSON Schema** ([`zarr_metadata.json_schema`](api/json_schema.md)): the
- JSON Schema of a `TypedDict`, a v3 declaration, or a whole `zarr.json`.
-- **Optional Pydantic integration** ([`zarr_metadata.pydantic`](api/pydantic.md),
- requires Pydantic 2.13 or newer): each model as a Pydantic field type that
- validates raw documents through the same strict parser.
-
-## What this is for
-
-- **Static Type Checking** The public `TypedDict` definitions describe the static JSON shape of Zarr
- metadata.
-
-- **Dynamic JSON validation** For strict, location-aware validation of JSON loaded from disk, use the
- model parser:
-
- ```python test="true" session="index"
- import json
- from zarr_metadata.model import ZarrV3ArrayMetadata
-
- with open("zarr.json", "rb") as f:
- raw = json.load(f)
-
- metadata = ZarrV3ArrayMetadata.from_json(raw)
- ```
-
- The optional Pydantic integration delegates raw input to the same strict
- parser and returns the same normalized model class:
-
- ```python test="true" session="index"
- from pydantic import TypeAdapter
- import zarr_metadata.pydantic as zmp
-
- adapter: TypeAdapter[zmp.ZarrV3ArrayMetadataField] = TypeAdapter(zmp.ZarrV3ArrayMetadataField)
- metadata = adapter.validate_python(raw)
- encoded = metadata.to_key_value()["zarr.json"]
- ```
-
- A field annotated `Annotated[ZarrV3ArrayMetadata, zmp.InScope(CONTEXT)]`
- is read in `CONTEXT`, whatever the validation context holds, and a model
- instance given to it is read again in `CONTEXT`.
-
- Note that a bare `TypeAdapter` over a public `TypedDict` is a **coercive** shape
- adapter, not a Zarr conformance validator; it may coerce values or discard
- members that the strict model parser rejects. This is better suited for handling input provided
- from Python objects (e.g., user-authored dicts) than JSON.
-
-## Validation boundary
-
-The readers and model constructors enforce the declared document structure and a small set
-of context-free consistency rules, e.g.:
-- fixed format literals,
-- finite JSON numbers outside `attributes` and the members nothing in scope reads
-- non-negative dimensions
-- one `dimension_names` entry per array dimension.
-
-In a v3 document, validation also checks each extension point: the data
-type, the chunk grid, the chunk key encoding, each codec, and each storage
-transformer. Each one is checked by the definition that claims its name in
-a scope. The scope is `CORE_AND_EXTENSIONS_CONTEXT` unless a `context` is passed.
-`ZARR_PYTHON_CONTEXT` adds the data types that zarr-python writes but no
-specification registers.
-If the definition rejects a configuration, the reader rejects it too,
-and a configuration key the definition does not declare is reported as
-`unknown_key`. If no definition in the scope claims a name, the reader
-does not check that extension point, and the consumer decides whether to
-support it.
-
-Making the context and explicit part of metadata parsing allows different libraries, which might support different
-sets of Zarr features, to formally represent exactly the codecs, data types, etc they support when
-parsing metadata.
-
-Some checks relate one part of a v3 document to another:
-
-- The fill value is checked against the data type the document names, by
- that data type's definition.
-- The chunk grid is checked against the shape, by the grid's definition.
-- The codecs are checked as a pipeline. In order, each codec's definition
- checks the codec against the chunk it receives. This includes a shard's
- inner codecs and index codecs.
-
-The readers do no arithmetic on values. For example, they do not check
-whether a fill value survives a `cast_value` round trip.
-
-Some choices that the spec text leaves open, or answers in two ways:
-
-- **`attributes` may hold `NaN`, `Infinity` and `-Infinity`.** The spec
- does not interpret attributes, and zarr-python and xarray write these
- numbers there (for example, a CF `_FillValue`). The models read them,
- and `to_key_value` writes them back as bare tokens, which a strict JSON
- parser rejects. So may a member the spec does not define when no
- definition in scope claims it: nothing reads it, so it is kept as
- written. `check`, from `zarr_metadata.typed_json`, rejects a non-finite
- number anywhere, including in attributes.
-- **A reader walks at most 256 levels of nesting.** A value nested deeper
- is an `invalid_value` problem at the first level past the limit,
- wherever it is. Every reader, writer and comparison uses one stack
- frame per level, and `copy.deepcopy` (and `pickle` before Python 3.12)
- uses two. A document at the limit therefore uses about half of the
- interpreter's default recursion limit, and the rest is left for the
- caller.
-- **`consolidated_metadata: null` is a problem.** A zarr-python 3.0.x bug
- wrote it. The spec requires an object, and the package accepts only an
- object. A reader of those stores removes the key before reading.
-- **An extension name must follow the spec's naming rule**,
- `^[a-z][a-z0-9-_.]+$`, or be a URI, which earlier versions of the spec
- required. Any other name is rejected before any definition is
- consulted, so `""` and `"foo/bar"` are problems, not unknown extensions.
-- **`must_understand: false` is rejected at every extension point**,
- including codecs and storage transformers, although the core spec names
- only the data type, chunk grid and chunk key encoding. A reader that
- skips a codec reads wrong bytes, just as a reader that skips a data type
- reads wrong values. On an unknown top-level member, `must_understand`
- keeps its meaning: a reader can skip that member.
-- **A v2 datetime or timedelta dtype must have a unit**, as in ` zarr_metadata.definition': api/definition.md
diff --git a/packages/zarr-metadata/tests/test_docs.py b/packages/zarr-metadata/tests/test_docs.py
index 7726151126..9c1d72122b 100644
--- a/packages/zarr-metadata/tests/test_docs.py
+++ b/packages/zarr-metadata/tests/test_docs.py
@@ -71,3 +71,31 @@ def test_every_python_block_says_whether_it_runs() -> None:
)
]
assert unmarked == []
+
+
+def test_the_extensions_page_example_is_the_definition_module_docstring_example() -> None:
+ """The acme.lz4 example on the extensions page is, line for line, the example in the `zarr_metadata.definition` module docstring. Only the page's copy runs under these tests, so this is what keeps the docstring's copy from drifting."""
+ import textwrap
+
+ import zarr_metadata.definition
+
+ page_blocks = [
+ example
+ for example in _examples()
+ if example.path == PACKAGE_ROOT / "docs" / "extensions.md"
+ and example.prefix_settings().get("session") == "extensions"
+ ]
+ assert len(page_blocks) == 1, (
+ "expected exactly one extensions-session block on docs/extensions.md"
+ )
+ docstring = zarr_metadata.definition.__doc__
+ assert docstring is not None, "zarr_metadata.definition has no module docstring"
+ lines = docstring.splitlines()
+ marker = "from collections.abc import Iterator"
+ starts = [i for i, line in enumerate(lines) if line.strip() == marker]
+ assert len(starts) == 1, f"expected one line {marker!r} in the docstring, found {len(starts)}"
+ end = starts[0]
+ while end < len(lines) and (lines[end].startswith(" ") or lines[end].strip() == ""):
+ end += 1
+ docstring_example = textwrap.dedent("\n".join(lines[starts[0] : end])).strip()
+ assert page_blocks[0].source.strip().splitlines() == docstring_example.splitlines()