IPC: support custom key/value metadata on messages (parity with Arrow IPC custom_metadata)

[rustyconover]

Problem or motivation

Vortex IPC has no way to attach application key/value metadata to a stream or to an individual message. Arrow IPC supports both: Schema.custom_metadata covers the whole stream and Message.custom_metadata covers a single record batch. Protocols built on Arrow IPC rely on this to send control information in-band next to the data.

One example is vgi-rpc, an RPC framework built on Arrow IPC with implementations in several languages. Every request and response batch carries metadata such as the method name, request ID, serialized stream state, cancellation signals, log records and error details (key definitions). Batches are handled as a RecordBatch paired with its metadata (AnnotatedBatch). Some messages are only metadata, sent as zero-row batches.

We'd like to evaluate Vortex IPC as an optional wire encoding for vgi-rpc. In a quick benchmark on a 1M-row trades-style table (sorted timestamps, low-cardinality symbols, 2-decimal prices), Vortex IPC came out at about 4.4 MB, compared with about 11 MB for Arrow IPC with zstd. Today, though, ArrayMessage only carries row_count and encodings. Without a metadata field we would have to put a second framing layer around every Vortex message, which defeats much of the point of using the format directly.

Proposed solution

Add an optional key/value list to the IPC Message table in vortex-ipc/flatbuffers/vortex-serde/message.fbs:

table KeyValue {
    key: string;
    value: [ubyte];   // or string, to mirror Arrow exactly
}

table Message {
    version: MessageVersion = V0;
    header: MessageHeader;
    body_size: uint64;
    custom_metadata: [KeyValue];   // new, optional
}

• Adding an optional table field is backward compatible in FlatBuffers. Existing readers ignore it, and existing streams decode as "no metadata".
• Putting it on Message rather than on ArrayMessage covers both cases: metadata on a DTypeMessage acts as stream-level metadata (Arrow's Schema.custom_metadata), and metadata on an ArrayMessage acts as per-batch metadata.
• On the API side:
  • MessageEncoder / ArrayStreamIPC would accept optional metadata per message.
  • DecoderMessage, SyncIPCReader and AsyncIPCReader would surface it, for example by yielding (ArrayRef, Option<Metadata>) or through a side accessor, so the existing ArrayStream item type doesn't change.
• Python bindings would expose it as a dict[bytes, bytes] / pa.KeyValueMetadata-compatible mapping.

A related gap: the Python bindings expose single-array IPC encode/decode (encode_ipc_array_buffers / decode_ipc_array_buffers), but not the SyncIPCReader / stream writer. A vortex.ipc.StreamReader(file_like) / StreamWriter(sink, dtype) pair would make streaming usable from Python. I'm happy to split that into a separate issue if you prefer.

Additional context

We're happy to contribute a PR for the flatbuffer and Rust changes, plus a Python stream reader/writer prototype that exercises them, if the maintainers agree with this direction.

[rustyconover]

Hi @mprammer this is part of me looking into using the Vortex IPC format.