Add public synchronous API for concurrent group/metadata operations #3859

Activity

1. d-v-b commented on Apr 1, 2026

   

   d-v-b

   on Apr 1, 2026

   Contributor

   More actions

   Thanks for this great summary!

   1. Public synchronous store API

   Zarr currently has no public synchronous API for store classes. PR #3638 added _get_bytes_sync, _get_json_sync as private methods on the Store ABC, with the intent to make them public once the API design matures (specifically the prototype parameter question). Promoting these to public would give downstream libraries a supported way to do concurrency-backed store operations.

   I think it would be great for these useful methods to be come public, and to add even more useful synchronous methods. These would be changes that are safe to make to our existing store classes. I think it would be even better if the synchronous methods were just called <method>, while the async methods were called <method>_async or equivalent, and if the store methods had useful return types (something more useful than a Buffer). Those changes are unsafe to make on our current store classes, and would require new store classes. The timeline for the additive, safe changes is much shorter than the timeline for a full refresh of our store API.

   2. Higher-level concurrent group opening

   As @d-v-b suggested in #3835, a synchronous function that opens multiple groups concurrently, async internally:

   Opens all child groups/arrays concurrently, returns sync objects
   children: dict[str, zarr.Array | zarr.Group] = zarr.open_members(group)

   This directly addresses the xarray/xradar open_datatree use case — fetching metadata and coordinate arrays for N groups in parallel. Internally it would use AsyncGroup.members() and the async store APIs, but callers wouldn't need to touch the event loop.

   IMO accessing members needs to go through an existing group for consolidated metadata to work properly, so we are looking at a new function / method on the Group class. Group.members is already in use, which is unfortunate because defining a NodeMembers object and using it e.g.Group.members.open_many(), Group.members.iter_arrays(), Group.members.iter_groups() would have had a nice shape. But we can settle for something here. One thing to fix first: we need to make separate group classes for v2 and v3 hierarchies. This will simplify internal sanity checks (to ensure that v3 groups never reveal v2 arrays). I would love to review a PR that adds this distinction.
2. aladinor commented on Apr 2, 2026

   

   aladinor

   on Apr 2, 2026

   Author

   More actions

   Thanks for the great discussion! After implementing the async open_datatree in xarray (pydata/xarray#10742), I want to share what we actually need from zarr and why higher-level APIs alone may not be sufficient.

   Two bottlenecks, not one

   Our xarray PR solves two distinct concurrency bottlenecks:

   Bottleneck 1: Group discovery + dataset opening

   • Uses async_root.members(max_depth=None) to walk the tree in one async pass
   • Opens each group's dataset concurrently via asyncio.TaskGroup
   • A zarr API like open_children() could help here

   Bottleneck 2: Concurrent index creation (the bigger win)

   • For each group, coordinate variables must be loaded to create indexes
   • Each coordinate load triggers a store.get() — doing this sequentially across N groups × M coordinates = O(N×M) round trips
   • We solve this by loading all coordinates across all groups concurrently via asyncio.gather + var.load_async()
   • Open to suggestions on how zarr could better support this pattern — but the index creation logic itself is inherently xarray-specific (Datasets, CF decoding, coordinate handling)

   Why sync() is the fundamental piece

   Both bottlenecks are solved by async code that is deeply xarray-specific — creating Datasets, building indexes, decoding CF variables. This logic can't be pushed down into zarr. What we need is a way to run xarray's async code on zarr's event loop from xarray's sync open_datatree():

   # xarray's open_datatree (sync) — submits async work to zarr's loop
   from zarr.core.sync import sync as zarr_sync
   return zarr_sync(self._open_datatree_from_stores_async(...))
   
   # Concurrent index creation across all groups
   results = zarr_sync(create_indexes_async())

   Using zarr's event loop (rather than creating our own) is critical because:

   • Avoids conflicting with zarr's internal loop management
   • Shares zarr's thread pool and concurrency settings
   • Is Jupyter-safe (zarr already handles the nested event loop case)

   Community convergence on sync()

   Multiple independent efforts have converged on needing sync():

   • @TomNicholas separated the two questions clearly: the ideal future async interface vs. "Can xarray rely on zarr.core.sync.sync() so we can merge PRs that enormously speed up xarray today?"

   • @BorisTheBrave built PoC Set variables async pydata/xarray#11171 — an independent PoC that uses zarr's sync() to parallelize variable writing, calling it "just lifting the [async] wrapper one level higher"

   • @shoyer noted we'd still need a way to invoke zarr's event loop from inside xarray

   Proposal

   I'd love to see zarr.core.sync.sync() promoted to public API (e.g., zarr.sync()). This is the minimal change that unblocks both our xarray PR and @BorisTheBrave's approach.

   Higher-level APIs like open_children() or public store methods are still valuable additions — but they complement sync(), they don't replace it.

   cc @d-v-b @shoyer @TomNicholas @BorisTheBrave @ilan-gold