Consider supporting RFC 9842 (Compression Dictionary Transport) dcz framing

[dfa1]

Background

RFC 9842 — Compression Dictionary Transport — defines how HTTP clients/servers negotiate and use external dictionaries with Brotli and Zstandard.

It has two distinct layers:

1. Wire format — dcz (Dictionary-Compressed Zstandard): a 40-byte fixed header, followed by a standard Zstandard frame compressed against that dictionary. Same idea for Brotli (dcb), out of scope here.
2. HTTP semantics — three new headers (Use-As-Dictionary, Available-Dictionary, Dictionary-ID), URL pattern matching (WHATWG URL Pattern) to decide which dictionary applies to which resource, dictionary storage/cache management (treated like cookies), HTTPS-only operation, and cross-origin/CORS protections. Tracked separately in Framework-agnostic model layer for Compression Dictionary Transport headers #92.

dcz header, exact layout (RFC 9842 §5)

• Magic: 8 fixed bytes — 0x5e 0x2a 0x4d 0x18 0x20 0x00 0x00 0x00. This is a Zstandard skippable frame magic + little-endian size field, so a dcz-unaware zstd decoder can skip it per the standard skippable-frame mechanism and still land on a valid frame.
• Hash: 32 bytes — full SHA-256 digest of the dictionary content, no truncation.
• Total: 40 bytes, fixed, once per compressed message (not per chunk).

Why the hash lives in the frame, not just the headers

RFC 9842 also carries a dictionary hash/id in the Available-Dictionary (request) and Dictionary-ID (response) headers (see #92). The in-frame hash isn't redundant with those — it does a different job:

• Negotiation vs. decoding. The headers are for negotiating whether a dictionary can be used before any bytes are sent. The in-frame hash is for decoding: once compressed bytes exist, the decompressor must self-verify/select the dictionary without needing the original HTTP transaction's headers.
• Caching/CDN replay. A cached response can be served later over a different connection than the one that negotiated it. The stored bytes must carry their own "which dictionary" fact, since the original headers may not be available or re-checked at replay time.
• Layering. The zstd decompressor itself shouldn't need any HTTP-header awareness — it's handed Content-Encoding: dcz bytes and must pick/verify the dictionary from the stream alone, same principle as any self-describing compression frame.
• Fail-fast correctness. Decompressing with the wrong dictionary produces garbage or errors; checking the embedded hash first is a cheap mandatory sanity check that doesn't depend on trusting a header an intermediary could have dropped or altered.

Practical upshot for this repo: layer (1) and layer (2) are genuinely independent, not just conceptually — the dcz codec here can be implemented and tested with zero knowledge of the header/URL-matching model in #92.

Proposal

Support layer (1) only, in a framework-agnostic way:

• A small dcz frame codec: wrap/unwrap the 40-byte header (magic + SHA-256 hash) around compressed bytes, reusing the existing dictionary compress/decompress APIs.
• On encode: compute SHA-256 of the dictionary, prepend magic + hash, then zstd-compress with the dictionary.
• On decode: parse the header, validate the hash against the dictionary in hand, then decompress with it.
• No HTTP dependency — pure byte-in/byte-out, consistent with the segment-first zero-copy API style already used elsewhere in the library.

Explicitly out of scope

• HTTP header negotiation (Use-As-Dictionary / Available-Dictionary / Dictionary-ID) — see Framework-agnostic model layer for Compression Dictionary Transport headers #92
• URL pattern matching / dictionary applicability rules — see Framework-agnostic model layer for Compression Dictionary Transport headers #92
• Dictionary cache/storage lifecycle ("treat like a cookie")
• Cross-origin/CORS checks, HTTPS-only enforcement

These are HTTP-client/server concerns and belong in whatever HTTP stack (Jetty, OkHttp, etc.) consumes zstd-java's dictionary compression as a primitive — not in this library.

Open question

Worth a lightweight sketch of the dcz codec API before committing to it, to confirm it fits naturally alongside the existing Zstd*Dictionary classes.

[dfa1]

Naming decision (recorded so we don't re-litigate this): no new module. This stays a package inside the core zstd module — io.github.dfa1.zstd.dcz — since the codec adds no new dependency (stdlib SHA-256 + existing dictionary APIs) and is deliberately named after the RFC's own wire-format token, not a description of purpose (purpose spans both HTTP negotiation in #92 and non-HTTP uses like NAS/archival storage, so a functional name would be wrong for one of those).

Full rationale: ADR 0016.