Discrepancy in numcodecs.shuffle classification as a BytesBytesCodec vs ArrayArrayCodec

[maxrjones]

numcodecs/numcodecs/zarr3.py

Line 224 in 506c89b

class Shuffle(_NumcodecsBytesBytesCodec, codec_name="shuffle"):

classifies the shuffle codec as a BytesBytesCodec. zarr-developers/zarr-extensions#2 proposes formalizing it as a ArrayArrayCodec. While this distinction is an implementation decision, it matters for using the codec with Zarr V3 and downstream libraries. I think it should be a ArrayArrayCodec.

[d-v-b]

I think the shuffle transformation should probably be bytesbytes? because it requires that the input be contiguous memory, which can't be guaranteed for arrayarray

[maxrjones]

I think the shuffle transformation should probably be bytesbytes? because it requires that the input be contiguous memory, which can't be guaranteed for arrayarray

👍

The reason I hoped it would be an ArrayArrayCodec is that Zarr-Python's ArrayV2Metadata accepts multiple codecs as filters but only one for a compressor. How would I approach creating V2 metadata equivalent to these codecs from V3 Metadata?

(
BytesCodec(endian=<Endian.little: 'little'>),
Shuffle(codec_name='numcodecs.shuffle', codec_config={'elementsize': 8}),
Zlib(codec_name='numcodecs.zlib', codec_config={'level': 5})
)

[d-v-b]

in zarr v2, there is no real difference between a filter and a compressor. conventionally a compressor is something that compresses its input, but this isn't enforced at all. So you could put those three codecs in filters, and leave compressor set to None, and it should work.

[d-v-b]

I think the shuffle transformation should probably be bytesbytes? because it requires that the input be contiguous memory, which can't be guaranteed for arrayarray

I should be more precise here: the actual shuffle algorithm doesn't require contiguous memory, only that the input is a sequence that can be iterated over in fixed-size increments. So for many arrays this could be a valid array-array codec, since it doesn't destroy the indexability of those arrays. One could plausibly have a bytes-bytes codec and a array-array codec that both do this exact same transformation. (And same for the identity function f(x) -> x)

FWIW I think "preserves array indexing" is what we really mean when we distinguish between array-array and bytes-bytes. I believe the "array-array", "array-bytes", "bytes-bytes" categories were all invented solely to support the sharding codec, which needs to accept array indices as input in order to perform a partial read. If chunk data has been jumbled in a way that doesn't preserve array indexing (which many, but not all, bytes-bytes transformations do), then sharding won't work.

Outside of that use case, the different codec categories are not really well-defined IMO, in the sense that same data transformation could be the basis of "different" array-array and bytes-bytes codecs

[maxrjones]

addressed by zarr-developers/zarr-extensions@bc6a858