diff --git a/packages/zarr-metadata/README.md b/packages/zarr-metadata/README.md index a01ed26842..4b3c1293fe 100644 --- a/packages/zarr-metadata/README.md +++ b/packages/zarr-metadata/README.md @@ -2,410 +2,44 @@ Documentation: -Basic tools for modelling Zarr metadata, with minimal dependencies. +Types, models and validators for Zarr metadata documents, with minimal +dependencies. `zarr-metadata` is developed in the [zarr-python repository](https://github.com/zarr-developers/zarr-python/tree/main/packages/zarr-metadata) -and released independently of `zarr` itself. Install it with: +and released independently of `zarr`. Install it with: ``` pip install zarr-metadata ``` -## Who needs this - -This library supports software that interacts with Zarr metadata documents. - ## What this is -This is *not* a full Zarr implementation. It's a collection of data structures and routines that -closely model the content of the Zarr specifications, such as: - -- **Typed JSON shapes** ([`zarr_metadata.v2`](https://zarr-metadata.readthedocs.io/en/latest/api/v2/) and - [`zarr_metadata.v3`](https://zarr-metadata.readthedocs.io/en/latest/api/v3/)): `TypedDict` definitions and - `Literal` aliases for the JSON documents specified by the - [Zarr v2](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html) and - [Zarr v3](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html) - specifications, plus types for - [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/) and a - few widely-used-but-unspecified entities (e.g. consolidated metadata). -- **Document models** ([`zarr_metadata.model`](https://zarr-metadata.readthedocs.io/en/latest/api/model/)): immutable - models of whole metadata documents, each one a document and the scope - it was read in, with readers that report every problem at its - location, and store-key (de)serialization. A document - produced by `to_json` shares no mutable state with the model that produced - it. -- **Definitions and scopes** ([`zarr_metadata.definition`](https://zarr-metadata.readthedocs.io/en/latest/api/definition/), - with each format's kinds in `zarr_metadata.v3.definition` and - `zarr_metadata.v2.definition`): each extension's configuration as a - `TypedDict` with its rules, and the scope that decides which definitions - read a document. `zarr_metadata.v3` and `zarr_metadata.v2` export every - definition and scope of their format, such as `BYTES_CODEC` and - `CORE_AND_EXTENSIONS_CONTEXT`. -- **JSON type checking** ([`zarr_metadata.typed_json`](https://zarr-metadata.readthedocs.io/en/latest/api/typed_json/)): - `check` checks a JSON value against any `TypedDict` and reports every - problem with its location. -- **Problems** ([`zarr_metadata.problems`](https://zarr-metadata.readthedocs.io/en/latest/api/problems/)): the problem - values every reader returns, and the error that carries them. -- **JSON Schema** ([`zarr_metadata.json_schema`](https://zarr-metadata.readthedocs.io/en/latest/api/json_schema/)): the - JSON Schema of a `TypedDict`, a v3 declaration, or a whole `zarr.json`. -- **Optional Pydantic integration** ([`zarr_metadata.pydantic`](https://zarr-metadata.readthedocs.io/en/latest/api/pydantic/), - requires Pydantic 2.13 or newer): each model as a Pydantic field type that - validates raw documents through the same strict parser. - -## What this is for - -- **Static Type Checking** The public `TypedDict` definitions describe the static JSON shape of Zarr - metadata. - -- **Dynamic JSON validation** For strict, loc-aware validation of JSON loaded from disk, use the - model parser: - - ```python test="true" session="readme" - import json - from zarr_metadata.model import ZarrV3ArrayMetadata - - with open("zarr.json", "rb") as f: - raw = json.load(f) - - metadata = ZarrV3ArrayMetadata.from_json(raw) - ``` - - The optional Pydantic integration delegates raw input to the same strict - parser and returns the same normalized model class: - - ```python test="true" session="readme" - from pydantic import TypeAdapter - import zarr_metadata.pydantic as zmp - - adapter: TypeAdapter[zmp.ZarrV3ArrayMetadataField] = TypeAdapter(zmp.ZarrV3ArrayMetadataField) - metadata = adapter.validate_python(raw) - encoded = metadata.to_key_value()["zarr.json"] - ``` - - A field annotated `Annotated[ZarrV3ArrayMetadata, zmp.InScope(CONTEXT)]` - is read in `CONTEXT`, whatever the validation context holds, and a model - instance given to it is read again in `CONTEXT`. - - Note that a bare `TypeAdapter` over a public `TypedDict` is a **coercive** shape - adapter, not a Zarr conformance validator; it may coerce values or discard - members that the strict model parser rejects. This is better suited for handling input provided - from Python objects (e.g., user-authored dicts) than JSON. - -## Validation boundary - -The readers and model constructors enforce the declared document structure and a small set -of context-free consistency rules, e.g.: -- fixed format literals, -- finite JSON numbers outside `attributes` and the members nothing in scope reads -- non-negative dimensions -- one `dimension_names` entry per array dimension. - -In a v3 document, validation also checks each extension point: the data -type, the chunk grid, the chunk key encoding, each codec, and each storage -transformer. Each one is checked by the definition that claims its name in -a scope. The scope is `CORE_AND_EXTENSIONS_CONTEXT` unless a `context` is passed. -`ZARR_PYTHON_CONTEXT` adds the data types that zarr-python writes but no -specification registers. -If the definition rejects a configuration, the reader rejects it too, -and a configuration key the definition does not declare is reported as -`unknown_key`. If no definition in the scope claims a name, the reader -does not check that extension point, and the consumer decides whether to -support it. - -Some checks relate one part of a v3 document to another: - -- The fill value is checked against the data type the document names, by - that data type's definition. -- The chunk grid is checked against the shape, by the grid's definition. -- The codecs are checked as a pipeline. In order, each codec's definition - checks the codec against the chunk it receives. This includes a shard's - inner codecs and index codecs. - -The readers do no arithmetic on values. For example, they do not check -whether a fill value survives a `cast_value` round trip. - -Some choices that the spec text leaves open, or answers in two ways: - -- **`attributes` may hold `NaN`, `Infinity` and `-Infinity`.** The spec - does not interpret attributes, and zarr-python and xarray write these - numbers there (for example, a CF `_FillValue`). The models read them, - and `to_key_value` writes them back as bare tokens, which a strict JSON - parser rejects. So may a member the spec does not define when no - definition in scope claims it: nothing reads it, so it is kept as - written. `check`, from `zarr_metadata.typed_json`, rejects a non-finite - number anywhere, including in attributes. -- **A reader walks at most 256 levels of nesting.** A value nested deeper - is an `invalid_value` problem at the first level past the limit, - wherever it is. Every reader, writer and comparison uses one stack - frame per level, and `copy.deepcopy` (and `pickle` before Python 3.12) - uses two. A document at the limit therefore uses about half of the - interpreter's default recursion limit, and the rest is left for the - caller. -- **`consolidated_metadata: null` is a problem.** A zarr-python 3.0.x bug - wrote it. The spec requires an object, and the package accepts only an - object. A reader of those stores removes the key before reading. -- **An extension name must follow the spec's naming rule**, - `^[a-z][a-z0-9-_.]+$`, or be a URI, which earlier versions of the spec - required. Any other name is rejected before any definition is - consulted, so `""` and `"foo/bar"` are problems, not unknown extensions. -- **`must_understand: false` is rejected at every extension point**, - including codecs and storage transformers, although the core spec names - only the data type, chunk grid and chunk key encoding. A reader that - skips a codec reads wrong bytes, just as a reader that skips a data type - reads wrong values. On an unknown top-level member, `must_understand` - keeps its meaning: a reader can skip that member. -- **A v2 datetime or timedelta dtype must have a unit**, as in ` Iterator[ValidationProblem]: + if "dictionary" in configuration and "dictionary_size" not in configuration: + yield ValidationProblem( + ("dictionary_size",), "a dictionary needs its size", "missing_key" + ) + + +ACME_LZ4 = CodecDefinition( + name="acme.lz4", + configuration=AcmeLz4Configuration, + kind="bytes_bytes", + size="dynamic", + rules=acme_lz4_rules, +) +CONTEXT = CORE_AND_EXTENSIONS_CONTEXT.update(codecs=[*CORE_AND_EXTENSIONS_CONTEXT.codecs.values(), ACME_LZ4]) +``` + +Pass `context=CONTEXT` to any reader or model constructor, and a document +that names `acme.lz4` is checked against this definition. + +Some rules for the pieces: + +- The `TypedDict` must be a `typing_extensions.TypedDict`, and must say + how undeclared keys are handled: `closed=True`, `extra_items=`, or + `closed=False`. An open-by-default `TypedDict` is refused, because it + would accept a misspelled key silently. +- Its members must be JSON shapes: `int`, `float`, `bool`, `str`, `None`, + `JSONValue`, a `Literal`, `tuple[T, ...]`, `tuple[T1, T2]`, a union, a + `TypedDict`, `Mapping[str, V]`, a `NewType` or a type alias. A number + may carry bounds from annotated-types (`Gt`, `Ge`, `Lt`, `Le`, + `Interval`). Any other `Annotated` metadata is refused when the + definition is built. +- The rules run only on a configuration within its bounds, so a rule can + rely on them. A rule reports the location of a problem, and the value + there becomes the problem's `input`. Show values in messages with + `zarr_metadata.problems.shown`, so they read like the package's own. +- Define each function at module top level. A model holds the definitions + that read it, so it can be pickled only if they can. + +A data type also declares its fill value (`fill_value`, +`fill_value_rules`, `fill_value_canonical`) and how its values are stored +(`storage`). A chunk grid declares which shapes it fits (`shape_rules`) and +its chunk lengths (`chunk_lengths`). A codec declares what it checks about +the chunk it receives (`chunk_rules`), what it hands on (`transition`), and +any pipelines it contains (`pipelines`). The +[`zarr_metadata.v3.definition`](api/v3/definition.md) docstring describes +each of these. diff --git a/packages/zarr-metadata/docs/index.md b/packages/zarr-metadata/docs/index.md index b139dfd307..50058a3041 100644 --- a/packages/zarr-metadata/docs/index.md +++ b/packages/zarr-metadata/docs/index.md @@ -1,411 +1,89 @@ # zarr-metadata -Basic tools for modelling Zarr metadata, with minimal dependencies. +Types, models and validators for Zarr metadata documents, with minimal +dependencies. `zarr-metadata` is developed in the [zarr-python repository](https://github.com/zarr-developers/zarr-python/tree/main/packages/zarr-metadata) -and released independently of `zarr` itself. Install it with: +and released independently of `zarr`. Install it with: ``` pip install zarr-metadata ``` -## Who needs this - -This library supports software that interacts with Zarr metadata documents. - ## What this is -This is *not* a full Zarr implementation. It's a collection of data structures and routines that -closely model the content of the Zarr specifications, such as: - -- **Typed JSON shapes** ([`zarr_metadata.v2`](api/v2.md) and - [`zarr_metadata.v3`](api/v3/index.md)): `TypedDict` definitions and - `Literal` aliases for the JSON documents specified by the - [Zarr v2](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html) and - [Zarr v3](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html) - specifications, plus types for - [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/) and a - few widely-used-but-unspecified entities (e.g. consolidated metadata). -- **Document models** ([`zarr_metadata.model`](api/model.md)): immutable - models of whole metadata documents, each one a document and the scope - it was read in, with readers that report every problem at its - location, and store-key (de)serialization. A document - produced by `to_json` shares no mutable state with the model that produced - it. -- **Definitions and scopes** ([`zarr_metadata.definition`](api/definition.md), - with each format's kinds in `zarr_metadata.v3.definition` and - `zarr_metadata.v2.definition`): each extension's configuration as a - `TypedDict` with its rules, and the scope that decides which definitions - read a document. `zarr_metadata.v3` and `zarr_metadata.v2` export every - definition and scope of their format, such as `BYTES_CODEC` and - `CORE_AND_EXTENSIONS_CONTEXT`. -- **JSON type checking** ([`zarr_metadata.typed_json`](api/typed_json.md)): - `check` checks a JSON value against any `TypedDict` and reports every - problem with its location. -- **Problems** ([`zarr_metadata.problems`](api/problems.md)): the problem - values every reader returns, and the error that carries them. -- **JSON Schema** ([`zarr_metadata.json_schema`](api/json_schema.md)): the - JSON Schema of a `TypedDict`, a v3 declaration, or a whole `zarr.json`. -- **Optional Pydantic integration** ([`zarr_metadata.pydantic`](api/pydantic.md), - requires Pydantic 2.13 or newer): each model as a Pydantic field type that - validates raw documents through the same strict parser. - -## What this is for - -- **Static Type Checking** The public `TypedDict` definitions describe the static JSON shape of Zarr - metadata. - -- **Dynamic JSON validation** For strict, location-aware validation of JSON loaded from disk, use the - model parser: - - ```python test="true" session="index" - import json - from zarr_metadata.model import ZarrV3ArrayMetadata - - with open("zarr.json", "rb") as f: - raw = json.load(f) - - metadata = ZarrV3ArrayMetadata.from_json(raw) - ``` - - The optional Pydantic integration delegates raw input to the same strict - parser and returns the same normalized model class: - - ```python test="true" session="index" - from pydantic import TypeAdapter - import zarr_metadata.pydantic as zmp - - adapter: TypeAdapter[zmp.ZarrV3ArrayMetadataField] = TypeAdapter(zmp.ZarrV3ArrayMetadataField) - metadata = adapter.validate_python(raw) - encoded = metadata.to_key_value()["zarr.json"] - ``` - - A field annotated `Annotated[ZarrV3ArrayMetadata, zmp.InScope(CONTEXT)]` - is read in `CONTEXT`, whatever the validation context holds, and a model - instance given to it is read again in `CONTEXT`. - - Note that a bare `TypeAdapter` over a public `TypedDict` is a **coercive** shape - adapter, not a Zarr conformance validator; it may coerce values or discard - members that the strict model parser rejects. This is better suited for handling input provided - from Python objects (e.g., user-authored dicts) than JSON. - -## Validation boundary - -The readers and model constructors enforce the declared document structure and a small set -of context-free consistency rules, e.g.: -- fixed format literals, -- finite JSON numbers outside `attributes` and the members nothing in scope reads -- non-negative dimensions -- one `dimension_names` entry per array dimension. - -In a v3 document, validation also checks each extension point: the data -type, the chunk grid, the chunk key encoding, each codec, and each storage -transformer. Each one is checked by the definition that claims its name in -a scope. The scope is `CORE_AND_EXTENSIONS_CONTEXT` unless a `context` is passed. -`ZARR_PYTHON_CONTEXT` adds the data types that zarr-python writes but no -specification registers. -If the definition rejects a configuration, the reader rejects it too, -and a configuration key the definition does not declare is reported as -`unknown_key`. If no definition in the scope claims a name, the reader -does not check that extension point, and the consumer decides whether to -support it. - -Making the context and explicit part of metadata parsing allows different libraries, which might support different -sets of Zarr features, to formally represent exactly the codecs, data types, etc they support when -parsing metadata. - -Some checks relate one part of a v3 document to another: - -- The fill value is checked against the data type the document names, by - that data type's definition. -- The chunk grid is checked against the shape, by the grid's definition. -- The codecs are checked as a pipeline. In order, each codec's definition - checks the codec against the chunk it receives. This includes a shard's - inner codecs and index codecs. - -The readers do no arithmetic on values. For example, they do not check -whether a fill value survives a `cast_value` round trip. - -Some choices that the spec text leaves open, or answers in two ways: - -- **`attributes` may hold `NaN`, `Infinity` and `-Infinity`.** The spec - does not interpret attributes, and zarr-python and xarray write these - numbers there (for example, a CF `_FillValue`). The models read them, - and `to_key_value` writes them back as bare tokens, which a strict JSON - parser rejects. So may a member the spec does not define when no - definition in scope claims it: nothing reads it, so it is kept as - written. `check`, from `zarr_metadata.typed_json`, rejects a non-finite - number anywhere, including in attributes. -- **A reader walks at most 256 levels of nesting.** A value nested deeper - is an `invalid_value` problem at the first level past the limit, - wherever it is. Every reader, writer and comparison uses one stack - frame per level, and `copy.deepcopy` (and `pickle` before Python 3.12) - uses two. A document at the limit therefore uses about half of the - interpreter's default recursion limit, and the rest is left for the - caller. -- **`consolidated_metadata: null` is a problem.** A zarr-python 3.0.x bug - wrote it. The spec requires an object, and the package accepts only an - object. A reader of those stores removes the key before reading. -- **An extension name must follow the spec's naming rule**, - `^[a-z][a-z0-9-_.]+$`, or be a URI, which earlier versions of the spec - required. Any other name is rejected before any definition is - consulted, so `""` and `"foo/bar"` are problems, not unknown extensions. -- **`must_understand: false` is rejected at every extension point**, - including codecs and storage transformers, although the core spec names - only the data type, chunk grid and chunk key encoding. A reader that - skips a codec reads wrong bytes, just as a reader that skips a data type - reads wrong values. On an unknown top-level member, `must_understand` - keeps its meaning: a reader can skip that member. -- **A v2 datetime or timedelta dtype must have a unit**, as in ` zarr_metadata.definition': api/definition.md diff --git a/packages/zarr-metadata/tests/test_docs.py b/packages/zarr-metadata/tests/test_docs.py index 7726151126..9c1d72122b 100644 --- a/packages/zarr-metadata/tests/test_docs.py +++ b/packages/zarr-metadata/tests/test_docs.py @@ -71,3 +71,31 @@ def test_every_python_block_says_whether_it_runs() -> None: ) ] assert unmarked == [] + + +def test_the_extensions_page_example_is_the_definition_module_docstring_example() -> None: + """The acme.lz4 example on the extensions page is, line for line, the example in the `zarr_metadata.definition` module docstring. Only the page's copy runs under these tests, so this is what keeps the docstring's copy from drifting.""" + import textwrap + + import zarr_metadata.definition + + page_blocks = [ + example + for example in _examples() + if example.path == PACKAGE_ROOT / "docs" / "extensions.md" + and example.prefix_settings().get("session") == "extensions" + ] + assert len(page_blocks) == 1, ( + "expected exactly one extensions-session block on docs/extensions.md" + ) + docstring = zarr_metadata.definition.__doc__ + assert docstring is not None, "zarr_metadata.definition has no module docstring" + lines = docstring.splitlines() + marker = "from collections.abc import Iterator" + starts = [i for i, line in enumerate(lines) if line.strip() == marker] + assert len(starts) == 1, f"expected one line {marker!r} in the docstring, found {len(starts)}" + end = starts[0] + while end < len(lines) and (lines[end].startswith(" ") or lines[end].strip() == ""): + end += 1 + docstring_example = textwrap.dedent("\n".join(lines[starts[0] : end])).strip() + assert page_blocks[0].source.strip().splitlines() == docstring_example.splitlines()