refactor(zarr-metadata)!: call an extension-point value a declaration

.github/workflows/test.yml

@@ -34,7 +34,7 @@ jobs:
 
     strategy:
       matrix:
-        python-version: ['3.12', '3.13', '3.14']
+        python-version: ['3.12', '3.13', '3.14', '3.15']
         dependency-set: ["minimal", "optional"]
         os: ["ubuntu-latest"]
         include:
@@ -101,7 +101,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ['3.12', "3.14"]
+        python-version: ['3.12', "3.14", "3.15"]
         dependency-set: ["upstream", "min_deps"]
         exclude:
           - python-version: "3.14"
@@ -201,6 +201,32 @@ jobs:
       run: |
         HATCH_ENV=test.py3.13-minimal just benchmark
 
+  examples:
+    name: Example scripts
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+      with:
+        fetch-depth: 0 # required for hatch version discovery when uv builds zarr
+        persist-credentials: false
+    - name: Set up Python
+      uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
+      with:
+        python-version: '3.13'
+        cache: 'pip'
+    - name: Install uv
+      uses: astral-sh/setup-uv@07a047e608db84b3d8c06cd9a7c7dcc9c355c18e # v10.3.2
+    - name: Install Hatch
+      run: python -m pip install hatch==1.16.5
+    - name: Install just
+      uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1
+      with:
+        repo: casey/just
+        version: 1.58.0
+    - name: Run Examples
+      run: |
+        HATCH_ENV=test.py3.13-minimal just test-examples
+
   test-complete:
     name: Test complete
 
@@ -209,7 +235,8 @@ jobs:
         test,
         test-upstream-and-min-deps,
         doctests,
-        benchmarks
+        benchmarks,
+        examples
       ]
     if: always()
     runs-on: ubuntu-latest

Justfile

@@ -64,6 +64,10 @@ hypothesis *args:
 doctest *args:
     hatch run doctest:pytest tests/test_docs.py -v "$@"
 
+# Run each example script against the local checkout, in its PEP 723 environment
+test-examples *args:
+    hatch run {{ quote(hatch_env) }}:pytest --run-examples tests/test_examples.py -v "$@"
+
 # Run the benchmark suite
 benchmark *args:
     hatch run {{ quote(hatch_env) }}:pytest --benchmark-enable tests/benchmarks "$@"

changes/4560.bugfix.md

@@ -0,0 +1,4 @@
+Numcodecs filters such as `Delta` and `FixedScaleOffset` now pass chunks of
+their declared shape to the next codec when writing. Before, the next codec
+received a flattened 1-D chunk, so chaining a filter with `TransposeCodec` or
+a 2-D image codec failed or lost data.

examples/README.md

@@ -19,7 +19,7 @@ examples/
 To add a new example:
 
 1. Create a new subdirectory: `examples/my_example/`
-2. Add your Python code: `examples/my_example/my_example.py`
+2. Add your Python code: `examples/my_example/my_example.py`. Start it with a [PEP 723](https://peps.python.org/pep-0723/) header that lists its dependencies, including `zarr`. `just test-examples` runs every example against your local checkout, and CI runs it on every pull request.
 3. Create documentation: `examples/my_example/README.md`
 4. Create a documentation page at `docs/user-guide/examples/my_example.md`. The documentation page should simply link to the `README.md` and the source code, e.g.:
 

examples/codec_pipeline_performance/codec_pipeline_performance.py

@@ -1,5 +1,5 @@
 # /// script
-# requires-python = ">=3.11"
+# requires-python = ">=3.12"
 # dependencies = [
 #   "zarr @ git+https://github.com/zarr-developers/zarr-python.git@main",
 #   "numpy",

examples/sharding_coalescing/sharding_coalescing.py

@@ -1,5 +1,5 @@
 # /// script
-# requires-python = ">=3.11"
+# requires-python = ">=3.12"
 # dependencies = [
 #   "zarr @ git+https://github.com/zarr-developers/zarr-python.git@main",
 #   "numpy",

packages/zarr-metadata/README.md

@@ -41,14 +41,14 @@ closely model the content of the Zarr specifications, such as:
   `TypedDict` with its rules, and the scope that decides which definitions
   read a document. `zarr_metadata.v3` and `zarr_metadata.v2` export every
   definition and scope of their format, such as `BYTES_CODEC` and
-  `CORE_AND_EXTENSIONS`.
+  `CORE_AND_EXTENSIONS_CONTEXT`.
 - **JSON type checking** ([`zarr_metadata.typed_json`](https://zarr-metadata.readthedocs.io/en/latest/api/typed_json/)):
   `check` checks a JSON value against any `TypedDict` and reports every
   problem with its location.
 - **Problems** ([`zarr_metadata.problems`](https://zarr-metadata.readthedocs.io/en/latest/api/problems/)): the problem
   values every reader returns, and the error that carries them.
 - **JSON Schema** ([`zarr_metadata.json_schema`](https://zarr-metadata.readthedocs.io/en/latest/api/json_schema/)): the
-  JSON Schema of a `TypedDict`, a v3 field, or a whole `zarr.json`.
+  JSON Schema of a `TypedDict`, a v3 declaration, or a whole `zarr.json`.
 - **Optional Pydantic integration** ([`zarr_metadata.pydantic`](https://zarr-metadata.readthedocs.io/en/latest/api/pydantic/),
   requires Pydantic 2.13 or newer): each model as a Pydantic field type that
   validates raw documents through the same strict parser.
@@ -104,7 +104,9 @@ of context-free consistency rules, e.g.:
 In a v3 document, validation also checks each extension point: the data
 type, the chunk grid, the chunk key encoding, each codec, and each storage
 transformer. Each one is checked by the definition that claims its name in
-a scope. The scope is `CORE_AND_EXTENSIONS` unless a `context` is passed.
+a scope. The scope is `CORE_AND_EXTENSIONS_CONTEXT` unless a `context` is passed.
+`ZARR_PYTHON_CONTEXT` adds the data types that zarr-python writes but no
+specification registers.
 If the definition rejects a configuration, the reader rejects it too,
 and a configuration key the definition does not declare is reported as
 `unknown_key`. If no definition in the scope claims a name, the reader
@@ -211,38 +213,39 @@ with one of them is refused:
 `read_array_metadata_v3` reads a document once and returns everything the
 read found:
 
-- each field as the scope read it: an `AcceptedField` when a definition
-  claims its name, an `UnclaimedField` when none does, or a
-  `RefusedField`, together with the field's location and the kind it was
+- each declaration as the scope read it: an `AcceptedDeclaration` when a definition
+  claims its name, an `UnclaimedDeclaration` when none does, or a
+  `RefusedDeclaration`, together with the declaration's location and the kind it was
   read as;
 - each codec, with the chunk it receives;
 - every problem;
 - the model, when there are no problems.
 
 `from_json` returns that model, or raises the problems. A consumer can
-apply its own policy by walking the fields, without reading anything
-twice. `with_problems` pairs each field with its problems: those located
-in the field and in the fields it contains. This matches how zod's
+apply its own policy by walking the declarations, without reading anything
+twice. `with_problems` pairs each declaration with its problems: those located
+in the declaration and in the declarations it contains. This matches how zod's
 `flattenError` groups issues. `canonical_of` returns the simplest
-spelling of a field that has no problems, without reading it again. For
-example, the code below finds the fields that go beyond the core spec (a
-field with no name is already a problem), and the simplest spelling of
-each field:
+spelling of a declaration that has no problems, without reading it again. For
+example, the code below finds the declarations that go beyond the core spec (a
+declaration with no name is already a problem), and the simplest spelling of
+each declaration:
 
 ```python test="true" session="readme"
 from zarr_metadata.model import read_array_metadata_v3
 from zarr_metadata.definition import canonical_of, with_problems
-from zarr_metadata.v3.definition import CORE
+from zarr_metadata.v3.definition import CORE_CONTEXT
 
 reading = read_array_metadata_v3(raw)
 beyond_core = [
     loc
-    for loc, field in reading.fields()
-    if field.name is not None and CORE.claimant(field.read_as, field.name) is None
+    for loc, declaration in reading.declarations()
+    if declaration.name is not None
+    and CORE_CONTEXT.claimant(declaration.read_as, declaration.name) is None
 ]
 simplest = {
-    loc: canonical_of(field, problems)
-    for loc, field, problems in with_problems(reading.fields(), reading.problems)
+    loc: canonical_of(declaration, problems)
+    for loc, declaration, problems in with_problems(reading.declarations(), reading.problems)
 }
 metadata = reading.metadata  # None when reading.problems is not empty
 ```
@@ -272,33 +275,34 @@ string against those rules.
 A top-level member that the spec does not define is an extra field. A
 scope can hold a definition for one, by its key, in the
 `array_extra_fields` or `group_extra_fields` slot of `ZarrV3Context`. A
-member whose key a definition claims is read by that definition, like any
-other field. Any other member is kept as written, as an `UnclaimedField`.
+member whose key a definition claims is read by that definition: an extra field is a declaration
+written as the member alone, `{key: value}`. Any other extra field is kept as
+written, as an `UnclaimedDeclaration`.
 The scope is what a reader recognizes, so the model's
 `must_understand_fields` lists each unclaimed member that does not say
 `"must_understand": false`: the spec requires a reader to fail to open a
-node with any such member. It covers members only. A data type, chunk
+node with any such member. It covers extra fields only. A data type, chunk
 grid, chunk key encoding, codec or storage transformer that nothing in
-scope claims is an `UnclaimedField` too, also when it sits inside another
-field, such as a codec inside a shard. A model's `unclaimed_fields` lists
-every field nothing in scope claims, at any depth, by its location, and
+scope claims is an `UnclaimedDeclaration` too, also when it sits inside another
+declaration, such as a codec inside a shard. A model's `unclaimed_declarations` lists
+every declaration nothing in scope claims, at any depth, by its location, and
 leaves out the members that waive `must_understand`: a reader can open
-the node when it is empty. `CORE` holds
+the node when it is empty. `CORE_CONTEXT` holds
 `CONSOLIDATED_METADATA_EXTRA_FIELD`, which reads a group's
 `consolidated_metadata` as the envelope the spec fixes. A group's
 `consolidated()` reads the documents that member lists, in the group's
 own scope.
 
 A v3 model consists of its document and the scope it was read in.
 `ZarrV3ArrayMetadata(document, context=None)` reads the document in the
-scope (`CORE_AND_EXTENSIONS` when none is given) and raises
+scope (`CORE_AND_EXTENSIONS_CONTEXT` when none is given) and raises
 `MetadataValidationError` with every problem, so an invalid model is never
 built. `to_json` returns the document as it was written, and
 `to_key_value` writes that same document to store bytes.
 
-Every typed member is a view of that read: each field as the scope read
-it (an `AcceptedField` or an `UnclaimedField`), and `shape`, `attributes`
-and the other members as the read refined them. These views are read-only
+Every typed attribute is a view of that read: each declaration as the scope read
+it (an `AcceptedDeclaration` or an `UnclaimedDeclaration`), and `shape`, `attributes`
+and the members that hold no declaration as the read refined them. These views are read-only
 at every level: a list in the document is shown as a tuple, and an object
 as a read-only mapping. `to_json` returns plain containers.
 
@@ -322,7 +326,7 @@ all of them with `ZarrV3ConsolidatedMetadata`, or with
 `read_consolidated_metadata_v3`, which returns problems instead of
 raising them. A document with a problem then fails only when it is read.
 Pass `context=group.context` to read the documents in the group's own
-scope; with no `context`, they are read in `CORE_AND_EXTENSIONS`.
+scope; with no `context`, they are read in `CORE_AND_EXTENSIONS_CONTEXT`.
 
 To write consolidated metadata, build a `ZarrV3ConsolidatedMetadata` and
 pass it to the group's `update`. The group reads the listed models in its
@@ -332,10 +336,10 @@ models as entries. Such an entry is accepted when its claims refine into
 the model's scope, and refused at its path otherwise. To consolidate
 children that were read in several scopes, consolidate them in the
 `union` of those scopes. Every reader takes `context=None` for the
-default scope, `CORE_AND_EXTENSIONS`.
+default scope, `CORE_AND_EXTENSIONS_CONTEXT`.
 
 Two models are equal when they represent the same document, even if each
-is spelled differently. Values the package interprets (each field, and the
+is spelled differently. Values the package interprets (each declaration, and the
 fill value against its data type) are compared by their canonical
 spelling, as returned by `canonical_of` and `canonical_fill_value`. For
 example, `"NaN"` and `"0x7fc00000"` are the same `float32` fill value,
@@ -355,7 +359,7 @@ the same data. The v3 spec defines its blosc codec, and says `noshuffle`
 ignores `typesize`. A v2 node with no `.zattrs` equals one with an
 empty `.zattrs`, as a v3 document with and without an empty
 `attributes` are equal. Values the package does
-not interpret (attributes, extra fields, and the configuration of a field
+not interpret (attributes, extra fields, and the configuration of a declaration
 that nothing in the scope claims) are compared as JSON text. JSON text
 distinguishes `true` from `1` and `-0.0` from `0.0`, and treats `NaN` as
 equal to itself. Equal models have equal hashes, but may write different
@@ -365,11 +369,11 @@ model holds nothing that can be changed in place.
 `node_metadata_json_schema_v3` writes the rules the readers apply as a
 JSON Schema (draft 2020-12). Use it in an editor that checks a `zarr.json`
 while you write it, or with a validator in another language. Each
-extension point is described as a field as its scope reads it: a claimed
+extension point is described as a declaration as its scope reads it: a claimed
 configuration follows its definition's TypedDict, including bounds, and a
 name that nothing in the scope claims may have any configuration. The fill
 value must be a valid fill value for the data type the document names.
-`field_json_schema_v3(kind, *, context=None)` writes the schema of one field, and
+`declaration_json_schema_v3(kind, *, context=None)` writes the schema of one declaration, and
 `json_schema` writes the schema of any TypedDict as `check` reads it. All
 three are in `zarr_metadata.json_schema`. A schema describes each member on its own, not the rules that
 relate members to each other, so a document the schema accepts may still

packages/zarr-metadata/changes/4574.bugfix.1.md

@@ -2,6 +2,6 @@
     writes it. A record's problem was located under a `fields` key the
     document does not have, at `("dtype", "fields", 0, 1)`; it is now at
     `("dtype", 0, 1)`, and carries the value found there as its `input`.
-    `fields()` places each record's type there too. A definition's
+    `declarations()` places each record's type there too. A definition's
     functions still find the record types in `nested` at
     `("fields", 0, 1)`, their place in the configuration.

packages/zarr-metadata/changes/4574.bugfix.7.md

@@ -1,4 +1,4 @@
-A definition's rules are located where the document writes the field. A
+A definition's rules are located where the document writes the declaration. A
     `bytes` codec written as `"bytes"` or `{"name": "bytes"}`, with values
     of several bytes, reports its missing `endian` at
     `codecs.N.configuration`, the configuration it does not write, as a

packages/zarr-metadata/changes/4574.bugfix.9.md

@@ -5,8 +5,8 @@
     problems in the merged document, so a `.zarray` that wrote `attributes`
     beside a `.zattrs` that was not an object gave two problems at
     `("attributes",)`, a location in neither file. An array node's reading
-    locates its fields there too: its new `at` is `(".zarray",)`, so
-    `fields()` gives `(".zarray", "dtype")` and `with_problems` pairs each
-    field with its problems. Its model reads the merged document, so the
-    model's own `reading` locates the field at `("dtype",)`, and no longer
+    locates its declarations there too: its new `at` is `(".zarray",)`, so
+    `declarations()` gives `(".zarray", "dtype")` and `with_problems` pairs each
+    declaration with its problems. Its model reads the merged document, so the
+    model's own `reading` locates the declaration at `("dtype",)`, and no longer
     equals the node reading.

packages/zarr-metadata/changes/4575.feature.md

@@ -0,0 +1 @@
+Added `zarr_metadata.testing.strategies`, Hypothesis strategies that draw valid Zarr v3 metadata declarations, behind a new `testing` extra (`pip install zarr-metadata[testing]`). There is one strategy for every codec, data type, chunk grid and chunk key encoding the package defines that holds no other declaration. `declarations` draws every envelope spelling the spec allows, or only the canonical one with `spellings="canonical"`. Codecs can be drawn for the `Chunk` they are handed, chunk grids for an array shape, and fill values for a data type. The strategies are a mapping keyed by definition, so an extension author can add strategies for their own definitions.

packages/zarr-metadata/changes/4583.feature.md

@@ -0,0 +1 @@
+Add the `ZARR_PYTHON_CONTEXT` scope, which reads what zarr-python writes: `CORE_AND_EXTENSIONS_CONTEXT` plus zarr-python's own data types `null_terminated_bytes` (NumPy's `S`), `raw_bytes` (NumPy's `V`) and `variable_length_bytes`, which no specification registers. `CORE_AND_EXTENSIONS_CONTEXT` now also reads the `fixed_length_utf32` data type (NumPy's `U`) and the `vlen-utf8` and `vlen-bytes` codecs that zarr-extensions registers, so the strings zarr-python writes are checked instead of read as unclaimed.

packages/zarr-metadata/changes/4583.removal.md

@@ -0,0 +1 @@
+Rename the scopes `CORE`, `CORE_AND_EXTENSIONS` and `CORE_V2` to `CORE_CONTEXT`, `CORE_AND_EXTENSIONS_CONTEXT` and `CORE_V2_CONTEXT`, so that a scope's name says it is a context. The old names, imported from `zarr_metadata.v3`, `zarr_metadata.v3.definition`, `zarr_metadata.v2` or `zarr_metadata.v2.definition`, still refer to the same scopes with a `DeprecationWarning`, and will be removed in a later release.

packages/zarr-metadata/changes/4588.feature.md

@@ -0,0 +1 @@
+`zarr_metadata.v3.data_type`, `zarr_metadata.v3.codec`, `zarr_metadata.v3.chunk_grid` and `zarr_metadata.v3.chunk_key_encoding` re-export every name their submodules export, so `from zarr_metadata.v3.data_type import BoolDataTypeName, Uint8DataTypeName` works again, as it did before 0.6.0. Each name is the same object as in its submodule, which stays its documented home.

packages/zarr-metadata/changes/4590.doc.md

@@ -0,0 +1 @@
+Adds a docstring and an API reference entry for every public object.

packages/zarr-metadata/changes/4592.removal.md

@@ -0,0 +1,10 @@
+Rename the word for a value at an extension point: it is now a *declaration* (it was a *field*). A name/value pair in a JSON metadata object stays a *member*. The package already said *field* in the JSON sense in `extra_fields`, `ExtraFieldDefinition`, `ZarrV3ExtensionField` and `must_understand_fields`, after the spec's "additional fields", so one word meant two things. The public names change with no aliases:
+
+- `AcceptedField`, `UnclaimedField`, `RefusedField` and `ResolvedField` are now `AcceptedDeclaration`, `UnclaimedDeclaration`, `RefusedDeclaration` and `ResolvedDeclaration`.
+- `ZarrV3DataTypeField`, `ZarrV3ChunkGridField`, `ZarrV3ChunkKeyEncodingField`, `ZarrV3CodecField`, `ZarrV3StaticCodecField`, `ZarrV3StorageTransformerField` and `ZarrV2DataTypeField` are now the same names ending in `Declaration`, and a kind's `field_aliases` is now `declaration_aliases`.
+- `ZarrV3MetadataFieldJSON` is now `ZarrV3DeclarationJSON`, and `field_json_schema_v3` is now `declaration_json_schema_v3`.
+- A reading's `fields()` is now `declarations()`, and a model's `unclaimed_fields` is now `unclaimed_declarations`.
+- The first parameter of `with_problems` is now `declarations` (it was `fields`).
+- The JSON Schemas that `node_metadata_json_schema_v3` and `declaration_json_schema_v3` write name each alias under `$defs` by its new name, so a `$ref` to `#/$defs/ZarrV3CodecField` is now `#/$defs/ZarrV3CodecDeclaration`, and likewise for the other kinds.
+
+Messages that said *field* or *metadata field* for an extension-point value now say *declaration*, as in "expected a declaration (string or extension object)". Prose that said *extra member* or *extension member* now says *extra field* and *extension field*, after the names. `extra_fields`, `ExtraFieldDefinition`, `ZarrV3ExtensionField`, `StructField` and the pydantic `...MetadataField` types keep their names.