Skip to content

feat(zarr-metadata)!: write store bytes through a JSONWriter - #429

Draft
d-v-b wants to merge 4 commits into
mainfrom
feat/zarr-metadata-json-writer-config
Draft

d-v-b wants to merge 4 commits into
mainfrom
feat/zarr-metadata-json-writer-config

Conversation

@d-v-b

@d-v-b d-v-b commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Agent summary

Breaking: to_key_value takes a writer instead of an indent keyword. A JSONWriter owns how a document becomes JSON bytes: it holds the settings (indent, sort_keys, ensure_ascii, trailing_newline), and its encode method does the writing. to_key_value also accepts a plain mapping of the settings, so to_key_value(indent=2) becomes to_key_value({"indent": 2}), with no import.

  • JSONWriter is a final, frozen dataclass in zarr_metadata.model._json_writer, exported from zarr_metadata and zarr_metadata.model. encode holds every choice about the bytes: the separators of Python's json module, non-finite numbers written as bare tokens (as zarr-python writes attributes), and lone surrogates written as escapes when ensure_ascii is off. The private dump_store_json helper is gone, so nothing else writes store JSON.
  • JSONWriterSchema is the closed TypedDict of the settings, every one optional. JSONWriterLike = JSONWriter | JSONWriterSchema is what each model's to_key_value(writer=...) accepts. JSONWriter.from_dict turns either into a writer (a writer passes through), and to_dict goes back.
  • The defaults are those of json.dumps, which the models already used, so a bare to_key_value() writes the same bytes as before.
  • A setting that json.dumps would accept but misread raises when the writer is built: indent=True (read as 1), a negative indent (read as 0), or a string indent that is not JSON whitespace (written verbatim, giving invalid JSON). A mapping key that is not a setting raises TypeError, so a misspelled setting is not ignored.
  • docs/models.md gains a "Writing store bytes" section with a tested example.

tests/model/test_json_writer.py checks, for all five model types and a range of writers given both as JSONWriter and as mappings, that each key's bytes are what json.dumps writes under the same settings, and that the store reads back to an equal model. Each refused value has its own test.

The design moved from an earlier JSONWriterConfig settings object, which a separate private function read, to one object that both holds the settings and encodes, and accepts a mapping so callers need not import it. The agent rebased onto main as one commit and had a separate agent review the redesign. That review found no correctness bugs, and its two actionable points (subclasses passing through from_dict, and gaps in the success cases) are fixed here.

Follow-up elsewhere

The ArrayIO branch (claude/arrayio-lazy-metadata-48d8e5, fork drafts #419/#420) calls to_key_value(indent=zarr_config.get("json_indent")) in store_metadata. Once this lands, that call becomes to_key_value({"indent": zarr_config.get("json_indent")}). This PR does not touch that branch.

🤖 Generated with Claude Code

d-v-b and others added 3 commits November 8, 2026 23:11
…cannot open (zarr-developers#4604)

* feat(zarr-metadata): add a reader that reports what a reader cannot open

`read_openable_node_metadata_v3` reads a v3 `zarr.json` as
`read_node_metadata_v3` does and adds a problem of the new kind
`unclaimed_declaration` for each declaration a reader must understand
that nothing in scope claims: an unknown codec, data type or chunk grid,
or an unknown top-level member without `"must_understand": false`. Its
reading has no problems exactly when a reader that understands the scope
can open the node.

The other readers keep accepting unclaimed declarations, so the format
stays open; their docstrings and the parsing guide now point to
`unclaimed_declarations` and the new reader. Prompted by refkv's report
that an unknown v3 member parses without error.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(zarr-metadata): number the changelog fragment for #440

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(zarr-metadata): number the unopenable-declarations fragment after upstream PR 4604

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… each spec claim (zarr-developers#4605)

* docs(zarr-metadata): split the landing page into guide pages and link each spec claim

Shrink index.md to a short pitch, one example and a reader-by-reader list
of where to go. Move the rest into guide pages: models and readers, what
the readers check, consolidated metadata, repairing known writer bugs,
writing an extension, JSON Schema and Pydantic. Each claim backed by a
Zarr spec links to the spec section that makes it; the consolidated
metadata page says what the specs do and do not settle and links the
zarr-specs PRs and issues and the original zarr-python implementation.
Trim the README so it no longer mirrors the landing page, and say
"context" where the pages said "scope".

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(zarr-metadata): number the guide-pages fragment after its fork PR

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(zarr-metadata): say "member" for a JSON name/value pair in the guide pages

The rename branch keeps "member" for a name/value pair of a metadata
object, so the new pages follow: "a top-level member that the spec does
not define is an extra field". "Extra field" and "extension field" stay,
after the names extra_fields and ZarrV3ExtensionField.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(zarr-metadata): say "member" for a JSON name/value pair on every page

Sweep the parsing-validation and context pages, which said "field" for a
name/value pair of a metadata object, and state the one exception on the
models page: "extra field" follows the spec's "additional fields" and
the names extra_fields, ExtraFieldDefinition and must_understand_fields;
"field" otherwise means a field of a Python class, or a part of a
structured data type.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(zarr-metadata): say "member" in the two must_understand_fields sentences

The models page said "each unclaimed field" and "any such field" where the
landing page and every neighboring sentence say "member". Roborev flagged
the mismatch.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(zarr-metadata): say why the additional-fields link points at the latest spec

v3.1.0 has no "Additional fields" heading; PR 373 added it after the
release, so the link cannot be pinned. Say so beside the link.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(zarr-metadata): split the additional-fields sentence and date its claim

Roborev found the three-clause sentence hard to parse and its "no tagged
release yet" unanchored in time. Two sentences now, anchored to v3.1.0,
with a note to pin the link once a release carries the heading.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(zarr-metadata): fix six wording and example issues roborev found on the guide pages

The pydantic example now builds the manifest with model_validate, so it
type-checks as the next sentence claims. The consolidated page says three
other readers take a context, not two. The equality section quotes the v2
spec on a missing .zattrs in two sentences. The two numbered Sphinx
anchors are replaced by line ranges in the spec source at the v3.1.0 tag.
The landing page says once that the library does no chunk or store I/O.
The extensions page example and the zarr_metadata.definition docstring
example are pinned to each other by a test.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* test(zarr-metadata): make the example-drift test say what it could not find

Roborev asked for diagnostics: the test now asserts there is exactly one
extensions-session block and exactly one marker line in the docstring,
each with a message, and compares line lists so a mismatch points at the
differing line.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(zarr-metadata): build the extension example with union and say "context" throughout

On main a Context has no update method, so the acme.lz4 example on the
extensions page now adds its definition with union, as the definition
module docstring does. Two leftover "scope"s in the parsing page say
"context", and the models page says "member" for an extra field. The
example-drift test imports at module level.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(zarr-metadata): number the guide-pages fragment after upstream PR 4605

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(zarr-metadata): fence the module docstring examples as python so the API pages highlight them

The examples in the definition, json_schema, pydantic and typed_json
module docstrings were indented code blocks. An indented block names no
language and pymdownx.highlight does not guess one, so the API reference
rendered them as plain text. Each is now a ```python fence, and the
example-drift test reads the definition docstring's fence.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(zarr-metadata): write the guide pages in short sentences and lists

Long comma-chained sentences on the guide pages, README and landing page
are split into short sentences, and enumerations become bulleted or
numbered lists. models, validation-rules, consolidated-metadata, repair,
extensions, json-schema and pydantic are restructured; context and
parsing-validation keep their narrative voice and only have their long
sentences split. No link, claim or code block changes.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
`to_key_value` takes a writer instead of an `indent` keyword. A
`JSONWriter` owns how a document becomes JSON bytes: it holds the
settings (`indent`, `sort_keys`, `ensure_ascii`, `trailing_newline`)
and its `encode` method does the writing, so the separators, the bare
non-finite tokens and the surrogate escapes are no longer split off in
a private helper. `to_key_value` also accepts a mapping of the settings,
`JSONWriterSchema`, so callers need not import the class:
`to_key_value({"indent": 2})`.

The defaults are those of `json.dumps`, so a bare `to_key_value()` writes
the same bytes as before. A setting `json.dumps` would misread raises,
and so does a mapping key that is not a setting.

Rebased onto main as one commit, replacing the JSONWriterConfig design.

Assisted-by: ClaudeCode:claude-opus-5-5
@d-v-b
d-v-b force-pushed the feat/zarr-metadata-json-writer-config branch from 8771582 to 95857ac Compare November 9, 2026 12:31
@d-v-b d-v-b changed the title feat(zarr-metadata)!: replace to_key_value's indent keyword with JSONWriterConfig feat(zarr-metadata)!: write store bytes through a JSONWriter Nov 9, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant