Skip to content

Implement streaming decompression#26

Open
LordMike wants to merge 12 commits into
OpenDisplay:mainfrom
LordMike:streaming-zlib-vendor-uzlib
Open

Implement streaming decompression#26
LordMike wants to merge 12 commits into
OpenDisplay:mainfrom
LordMike:streaming-zlib-vendor-uzlib

Conversation

@LordMike
Copy link
Copy Markdown
Contributor

@LordMike LordMike commented May 8, 2026

Initial implementation of streaming decompression, written by OpenAI Codex. This is following my suggestion in #25, and the following benchmarks of compression, where zlib stands out as good. To combat the memory requirements though, we need to use a smaller Window Size, and we need to restructure the uzlib api a bit to support it. No API we could use, has a low state size while also offering streaming APIs, so we've adapted uzlib to get that.

  • Pulls in uzlib as code, rather than a github reference
  • Patches uzlib a lot, removing the one-shot API in favor of api methods that mimic heatshrink's API: od_zlib_stream_push, od_zlib_stream_poll, ..
  • Replaces the decompression of both the compressed full image and compressed partial image writing, with this new API
  • Reduces memory for decompression to about 2 KB (for 512 byte window size):
    • 1192 bytes for the state
    • 256 bytes for the output buffer (when reading from our decompressed stream), controlled by OPENDISPLAY_DECOMPRESSION_CHUNK_SIZE
    • 512 to 32768 bytes for window size, controlled by OPENDISPLAY_ZLIB_WINDOW_BITS

TODO

  • Find a way to signal support for smaller window sizes. Adding a new capability flag like ZIP, like STREAMING-ZIP, would signal clients that we support 512 byte window sizes.
    • Clients could also infer that ZIP + STREAMING-ZIP means any window size is possible.
    • If a device only offers STREAMING-ZIP, we know it supports deflate, but only at a 512 byte window size. Old clients will not understand STREAMING-ZIP, and will assume compression is not possible (as today)
  • Tweak py-opendisplay to support the above, and then use windowSize=9 when compressing for STREAMING-ZIP-only devices; and to not early exit if the compressed bytes would be too large.
  • Decide on which build targets get which build constants, and if some MCUs now support zip, where they didn't before.
  • Remove constants like TARGET_LARGE_MEMORY (?)

Fixes #25

@LordMike
Copy link
Copy Markdown
Contributor Author

We've set on reusing ZIPXL as the config capability flag to indicate streaming deflates. It signals roughly the same concept, as this streaming zip supports indefinitely large streams. The firmware early exits if a client sends a compressed stream with a too large windowsize, so callers will know somethings wrong.

I've also opted to let all MCUs get the default OPENDISPLAY_ZLIB_WINDOW_BITS = 9, meaning a window of 512 bytes. There was no real gain with larger, and then we have the benefit of not having to communicate window sizes.

@LordMike LordMike marked this pull request as ready for review May 22, 2026 20:37
@LordMike LordMike requested a review from jonasniesner as a code owner May 22, 2026 20:37
@LordMike
Copy link
Copy Markdown
Contributor Author

I reviewed the compatibility matrix for old clients, old firmware, new clients and new firmware - with both Zip and ZipXl exposed by the FW. I found that this PR would break all clients sending data, to any new firmware, if we keep the window size at 9 (512 bytes).

So I think the best of all worlds is for us to default to 9, but for all existing targets set it to 15. This allows existing clients to send data with the window size they have. New clients will use a smaller window size, which implicitly works for both old and new firmwares. New firmwares on new devices will use WS=9, which only works with new clients but thats likely ok, because these devices are new anyways, so we can assume a new client is used too.

This repurposes ZipXl to mean "streaming zip", and thats entirely fine. Because the end result is the same: Old users of zipxl work up to a limit of about 200 KB buffered data, with WS=15, which will work in the current MCU targets. New clients will assume ZipXl to mean "streaming zip, no limit", which will fail - but there are very few MCU's out there with ZipXl enabled today, and those users should be trusted to update the firmware when they update their clients too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support streaming decompression

1 participant