[v3] support for object arrays

[jhamman]

Zarr-Python 2 supported object arrays. This functionality has not made it into Zarr-Python 3 yet (in part because there is not an obvious way to develop a v3 dtype for arbitrary Python objects).

An example demonstrating this functionality using Zarr-Python 2:

z = zarr.empty(5, dtype=object, object_codec=numcodecs.JSON())
z[0] = 42
z[1] = 'foo'
z[2] = ['bar', 'baz', 'qux']
z[3] = {'a': 1, 'b': 2.2}
z[:]
array([42, 'foo', list(['bar', 'baz', 'qux']), {'a': 1, 'b': 2.2}, None], dtype=object)

This issue tracks the development of object array support in Zarr-Python 3.

[basnijholt]

I ran into this issue in pipefunc/pipefunc#523

in part because there is not an obvious way to develop a v3 dtype for arbitrary Python objects

Do you expect that object arrays will be supported at some early v3.* release?

[d-v-b]

my main concern with the object dtype is the danger associated with using pickle, or any other encoding of python objects that could result in arbitrary code execution. but I don't think we have reached a formal decision on object arrays in v3.

[basnijholt]

I am aware of that limitation/issue. In pipefunc we register our codec that uses cloudpickle.

[d-v-b]

so we chatted about this in the developer meeting, the conclusion was that supporting object dtype arrays directly is not in-scope for zarr-python 3.x, because of security concerns inherent to storing arbitrary python objects, and our commitment to keep zarr a format that's accessible to a wide range of languages.

that being said, we would be interested in identifying how zarr-python 3.x could be extended in a third party library to add features like an object dtype. Our dtypes today are not extensible, but I think this could be fixed, but this would require some design work first. Is that process something you would be interested in?

[mavaylon1]

@d-v-b Hi there. The NWB team would be interested in the idea being able to extend zarr-python to add object dtype. We would also be happy to work on this. You mentioned that currently we are not able to extend to create new dtypes?

[d-v-b]

correct, we haven't put together an API for user-defined dtypes in zarr python 3 yet. We definitely intend to add this feature, and we have a very promising proposal here: #2750. But I can't give you a definite timeline for when this feature would be released.