docs(zarr-metadata): split the landing page into guide pages and link each spec claim

packages/zarr-metadata/changes/434.doc.md

@@ -0,0 +1 @@
+The documentation now has a short landing page that points each kind of reader to a guide page, instead of one long page. New guide pages cover models and readers, the rules the readers check, consolidated metadata, repairing known writer bugs, writing an extension, JSON Schema and Pydantic. Each claim the pages make about a Zarr specification links to the section of the spec that makes it, and the consolidated metadata page says what the specs do and do not settle, with links to the zarr-specs pull requests and issues and to the original zarr-python implementation. The pages say "context" for a `Context` where some said "scope".

packages/zarr-metadata/docs/api/index.md

@@ -9,7 +9,7 @@ The package is organized to mirror the structure of the Zarr specifications:
 - [`zarr_metadata.json_schema`](json_schema.md) — JSON Schema output for
   any `TypedDict`, one v3 declaration, or a whole v3 `zarr.json`
 - [`zarr_metadata.model`](model.md) — document models (each one a
-  document and the scope it was read in), the readers that return
+  document and the context it was read in), the readers that return
   problems instead of raising them, and the `UNSET` sentinel
 - [`zarr_metadata.problems`](problems.md) — `ValidationProblem`,
   `ProblemKind`, `Problems` and `MetadataValidationError`, which every
@@ -21,7 +21,7 @@ The package is organized to mirror the structure of the Zarr specifications:
   typing spec defines them, with every problem located
 - [`zarr_metadata.v2`](v2.md) — `TypedDict` shapes for Zarr v2 documents
   (`.zarray`, `.zgroup`, `.zattrs`, `.zmetadata`), and
-  `zarr_metadata.v2.definition`: the v2 kinds of metadata, the scope
+  `zarr_metadata.v2.definition`: the v2 kinds of metadata, the context
   `CORE_V2_CONTEXT`, and the readers of one `dtype` or codec. Each dtype family's
   definition is in `zarr_metadata.v2.data_type`, and each codec's in
   `zarr_metadata.v2.codec`
@@ -32,10 +32,11 @@ The package is organized to mirror the structure of the Zarr specifications:
 - [`zarr_metadata.definition`](definition.md) — what both formats share
   for reading a declaration: `Definition`, `resolve`, the three results of a
   reading (`AcceptedDeclaration`, `UnclaimedDeclaration`, `RefusedDeclaration`), and the
-  scope type `Context`. Its module docstring is the guide
+  context type `Context`. Its module docstring is the guide
 - [`zarr_metadata.v3.definition`](v3/definition.md) — the five v3 kinds
   of metadata (`CodecDefinition`, `DataTypeDefinition`, ...), the codec
-  pipeline types, and the v3 scopes `CORE_CONTEXT` and `CORE_AND_EXTENSIONS_CONTEXT`
+  pipeline types, and the v3 contexts `CORE_CONTEXT`,
+  `CORE_AND_EXTENSIONS_CONTEXT` and `ZARR_PYTHON_CONTEXT`
 
 The document types, models, and spec vocabulary — including the store keys —
 are re-exported at the top level, so

packages/zarr-metadata/docs/consolidated-metadata.md

@@ -0,0 +1,95 @@
+# Consolidated metadata
+
+Consolidated metadata is a copy of every metadata document below a group,
+kept in one place so that a reader can open a hierarchy with one read. It
+is a convention of zarr-python, the reference implementation, and neither
+Zarr specification defines it in full. This page says what is settled
+before it describes the API.
+
+## Status in the specifications
+
+- **Zarr v2.** The v2 spec has no consolidated metadata. The `.zmetadata`
+  file was designed and shipped in zarr-python
+  [PR 268](https://github.com/zarr-developers/zarr-python/pull/268), merged in November 2018 and released in
+  zarr-python 2.3.0, and is described in the
+  [zarr-python 2 tutorial](https://zarr.readthedocs.io/en/v2.18.3/tutorial.html#consolidating-metadata).
+  zarr-specs [issue 113](https://github.com/zarr-developers/zarr-specs/issues/113), on standardizing the
+  format, is open.
+- **Zarr v3.** zarr-specs [PR 309](https://github.com/zarr-developers/zarr-specs/pull/309), open since
+  August 2024 and following [issue 136](https://github.com/zarr-developers/zarr-specs/issues/136), proposes the
+  `consolidated_metadata` member with the `kind: "inline"` envelope that
+  zarr-python 3 writes. It is not merged. The core spec does not define the
+  member's entries. Since [PR 373](https://github.com/zarr-developers/zarr-specs/pull/373), merged in July 2026,
+  it names the member "for historical reasons" and fixes its envelope: an
+  object with `must_understand: false`, `kind: "inline"` and a `metadata`
+  object ([spec text](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L802-L816)). That change closed
+  [issue 371](https://github.com/zarr-developers/zarr-specs/issues/371), which found that the v3.1.0 extension
+  rules, read strictly, put every consolidated group zarr-python 3 had
+  written out of spec, because the envelope has no `name`.
+
+This package reads the envelope the spec fixes, and the entries as
+zarr-python writes them.
+
+## Zarr v3
+
+Consolidated metadata holds the hierarchy below its group, with the group
+as the root. The document of the node at `/a/b` is stored under the key
+`a/b`, as the [stored representation](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#stored-representation) strips
+the leading `/` of a path. The documents and the group must form a tree:
+only groups have child nodes, and the parent of every node is present.
+`ZarrV3ConsolidatedMetadata` checks this when it reads the documents.
+
+`NodeName` and `NodePath`, in `zarr_metadata.v3`, are the types of strings
+that satisfy the spec's rules for [node names](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#node-names) and
+paths. They are modeled on the zarrs types with the same names.
+`validate_node_name_v3`, `is_node_name_v3` and `parse_node_name_v3`, and
+the matching `node_path` functions, check a string against those rules.
+
+### Reading
+
+A group does not read the documents its `consolidated_metadata` lists.
+Consolidated metadata is a convention of the reference implementation, not
+core metadata, so a group holds the member as written, after it checks the
+[envelope that the core spec fixes](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L802-L816). `CORE_CONTEXT` holds
+`CONSOLIDATED_METADATA_EXTRA_FIELD`, the definition that reads a group's
+`consolidated_metadata` as that envelope.
+
+A reader can use the member as a source of child documents in place of the
+store: read one document with `read_node_metadata_v3` when that node is
+opened, or read all of them with the group's `consolidated()`, with
+`ZarrV3ConsolidatedMetadata`, or with `read_consolidated_metadata_v3`,
+which returns problems instead of raising them. A document with a problem
+then fails only when it is read. `consolidated()` reads the documents in
+the group's own context. The other three take a `context`; pass
+`context=group.context` to read the documents in the group's own context,
+and with no `context` they are read in `CORE_AND_EXTENSIONS_CONTEXT`.
+
+### Writing
+
+To write consolidated metadata, build a `ZarrV3ConsolidatedMetadata` and
+pass it to the group's `update`. The group reads the listed models in its
+own context, refuses one that its context reads differently, and writes the
+document. The model reads every document strictly, and also accepts node
+models as entries. Such an entry is accepted when its claims refine into
+the model's context, and refused at its path otherwise. To consolidate
+children that were read in several contexts, consolidate them in the
+`union` of those contexts.
+
+## Zarr v2
+
+A v2 hierarchy's consolidated metadata is the `.zmetadata` document at the
+root of the store, as zarr-python 2.3 and later write it: an object with
+`zarr_consolidated_format: 1` and a `metadata` object that maps each
+metadata file's path to that file's contents, under keys such as
+`a/.zarray`, `a/.zattrs` and `a/b/.zgroup`. The format comes from
+zarr-python [PR 268](https://github.com/zarr-developers/zarr-python/pull/268); no specification defines it.
+
+`ZarrV2ConsolidatedMetadata` reads such a document, and
+`read_consolidated_metadata_v2` returns problems instead of raising them.
+Both take a `context`, `CORE_V2_CONTEXT` when none is given. Each `.zarray` or
+`.zgroup` entry is read as a node, merged with its sibling `.zattrs`. A
+`.zattrs` with no sibling is kept but forms no node, and any other entry
+must be JSON and is kept. `to_json` returns the entries keyed by file, as
+written. The `.zmetadata` that zarr-python 3 writes for a v2 hierarchy has
+a known bug, which `read_repaired_consolidated_metadata_v2` undoes; see
+[Repairing known writer bugs](repair.md).

packages/zarr-metadata/docs/context.md

@@ -4,7 +4,7 @@ Resolving metadata documents requires a context, a `Context`, that defines which
 
 ## Why do we need a context
 
-The Zarr V3 spec defines the JSON structure of `zarr.json` for arrays and groups exactly, by defining the fields that must be present and the JSON types they must have. But the __interpretation__ of those fields as actual data types, codecs, chunk grids, etc is not strictly defined by the spec. Relating a JSON value like `"bytes"` to the `bytes` codec requires a mapping from names to codec types.
+The Zarr V3 spec defines the JSON structure of `zarr.json` for arrays and groups exactly, by defining the members that must be present and the JSON types they must have. But the __interpretation__ of those members as actual data types, codecs, chunk grids, etc is not strictly defined by the spec. Relating a JSON value like `"bytes"` to the `bytes` codec requires a mapping from names to codec types.
 
 `zarr-metadata` makes this explicit by validating JSON documents with a __context object__. That context object contains definitions of codecs, data types, etc, which can be matched to the declarations in the JSON document.
 
@@ -47,7 +47,7 @@ document: ZarrV3ArrayMetadataJSON = {
 }
 ```
 
-In the above code snippet we define a `zarr.json` document with three codec declarations (we will just call them codecs for brevity): one named `bytes`, another named `gzip`, and a third named `unknown`. The `bytes` codec here is correctly configured according to the `bytes` codec spec. The `gzip` declaration here violates the `gzip` codec spec because the `level` configuration field is too high. And the `unknown` codec is unknown, it has no specification and cannot be resolved.
+In the above code snippet we define a `zarr.json` document with three codec declarations (we will just call them codecs for brevity): one named `bytes`, another named `gzip`, and a third named `unknown`. The `bytes` codec here is correctly configured according to the `bytes` codec spec. The `gzip` declaration here violates the `gzip` codec spec because the `level` configuration member is too high. And the `unknown` codec is unknown, it has no specification and cannot be resolved.
 
 These three codecs represent three different scenarios that this library handles, two with distinct failure modes. We demonstrate how this works in the next code snippet.
 
@@ -101,7 +101,7 @@ These are the three scenarios:
 
 - `bytes` is parsed to an `AcceptedDeclaration`. A definition in the context claims the name `bytes`, and the configuration passes its checks.
 - The malformed `gzip` is parsed to a `RefusedDeclaration`. A definition claims the name `gzip`, but the configuration fails one of its checks: a `level` of 99 is outside the range gzip allows. The read reports the problem, with its location in the document, and builds no model, so `reading.metadata` is `None`. Building the model directly raises the same problem as a `MetadataValidationError`.
-- The `unknown` codec is parsed to an `UnclaimedDeclaration`. Nothing in the context claims the name, so it is kept as written and not checked. This is not a problem: the spec lets a document name codecs a reader does not know.
+- The `unknown` codec is parsed to an `UnclaimedDeclaration`. Nothing in the context claims the name, so it is kept as written and not checked. This is not a problem: the spec [lets a document name codecs](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#extension-codecs) defined outside it, which a reader may not know.
 
 So one JSON document can be valid in one context and invalid in another. This is why a model is a document _and_ the context it was read in: the context defines the interpretation of the document. A model keeps its context, and moving it to another context reads the document again. Our `small` model was valid in its original context, but it is not valid in the new one:
 

packages/zarr-metadata/docs/extensions.md

@@ -0,0 +1,83 @@
+# Writing an extension
+
+A document may name a codec, data type, chunk grid, chunk key encoding or
+storage transformer that this package does not define: each is an
+[extension point](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#extension-points) of the v3 spec. The readers leave
+such a declaration unclaimed, and a program that needs to check it adds a
+**definition** to the context it reads in. The
+[`zarr_metadata.definition`](api/definition.md) module docstring is the
+full reference; this page is the short version.
+
+A definition is a value, not a class to subclass. It has a `name`, a
+`TypedDict` for the configuration's JSON, and, when the type cannot
+express every rule, a `rules` function that yields the spec violations in a
+configuration. A v3 codec also says what kind of codec it is and whether
+its output size is fixed by its input size.
+
+```python test="true" session="extensions"
+from collections.abc import Iterator
+from typing import Annotated, NotRequired
+
+from annotated_types import Ge
+from typing_extensions import TypedDict
+
+from zarr_metadata import ValidationProblem
+from zarr_metadata.definition import Nested
+from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition
+
+
+class AcmeLz4Configuration(TypedDict, closed=True):
+    acceleration: Annotated[int, Ge(1)]
+    dictionary: NotRequired[str]
+    dictionary_size: NotRequired[Annotated[int, Ge(1)]]
+
+
+def acme_lz4_rules(
+    configuration: AcmeLz4Configuration, nested: Nested
+) -> Iterator[ValidationProblem]:
+    if "dictionary" in configuration and "dictionary_size" not in configuration:
+        yield ValidationProblem(
+            ("dictionary_size",), "a dictionary needs its size", "missing_key"
+        )
+
+
+ACME_LZ4 = CodecDefinition(
+    name="acme.lz4",
+    configuration=AcmeLz4Configuration,
+    kind="bytes_bytes",
+    size="dynamic",
+    rules=acme_lz4_rules,
+)
+CONTEXT = CORE_AND_EXTENSIONS_CONTEXT.update(codecs=[*CORE_AND_EXTENSIONS_CONTEXT.codecs.values(), ACME_LZ4])
+```
+
+Pass `context=CONTEXT` to any reader or model constructor, and a document
+that names `acme.lz4` is checked against this definition.
+
+Some rules for the pieces:
+
+- The `TypedDict` must be a `typing_extensions.TypedDict`, and must say
+  how undeclared keys are handled: `closed=True`, `extra_items=`, or
+  `closed=False`. An open-by-default `TypedDict` is refused, because it
+  would accept a misspelled key silently.
+- Its members must be JSON shapes: `int`, `float`, `bool`, `str`, `None`,
+  `JSONValue`, a `Literal`, `tuple[T, ...]`, `tuple[T1, T2]`, a union, a
+  `TypedDict`, `Mapping[str, V]`, a `NewType` or a type alias. A number
+  may carry bounds from annotated-types (`Gt`, `Ge`, `Lt`, `Le`,
+  `Interval`). Any other `Annotated` metadata is refused when the
+  definition is built.
+- The rules run only on a configuration within its bounds, so a rule can
+  rely on them. A rule reports the location of a problem, and the value
+  there becomes the problem's `input`. Show values in messages with
+  `zarr_metadata.problems.shown`, so they read like the package's own.
+- Define each function at module top level. A model holds the definitions
+  that read it, so it can be pickled only if they can.
+
+A data type also declares its fill value (`fill_value`,
+`fill_value_rules`, `fill_value_canonical`) and how its values are stored
+(`storage`). A chunk grid declares which shapes it fits (`shape_rules`) and
+its chunk lengths (`chunk_lengths`). A codec declares what it checks about
+the chunk it receives (`chunk_rules`), what it hands on (`transition`), and
+any pipelines it contains (`pipelines`). The
+[`zarr_metadata.v3.definition`](api/v3/definition.md) docstring describes
+each of these.