Implement streaming decompression#26
Conversation
# Conflicts: # src/display_service.cpp # src/main.cpp # src/main.h
|
We've set on reusing ZIPXL as the config capability flag to indicate streaming deflates. It signals roughly the same concept, as this streaming zip supports indefinitely large streams. The firmware early exits if a client sends a compressed stream with a too large windowsize, so callers will know somethings wrong. I've also opted to let all MCUs get the default |
|
I reviewed the compatibility matrix for old clients, old firmware, new clients and new firmware - with both Zip and ZipXl exposed by the FW. I found that this PR would break all clients sending data, to any new firmware, if we keep the window size at 9 (512 bytes). So I think the best of all worlds is for us to default to 9, but for all existing targets set it to 15. This allows existing clients to send data with the window size they have. New clients will use a smaller window size, which implicitly works for both old and new firmwares. New firmwares on new devices will use WS=9, which only works with new clients but thats likely ok, because these devices are new anyways, so we can assume a new client is used too. This repurposes ZipXl to mean "streaming zip", and thats entirely fine. Because the end result is the same: Old users of zipxl work up to a limit of about 200 KB buffered data, with WS=15, which will work in the current MCU targets. New clients will assume ZipXl to mean "streaming zip, no limit", which will fail - but there are very few MCU's out there with ZipXl enabled today, and those users should be trusted to update the firmware when they update their clients too. |
Initial implementation of streaming decompression, written by OpenAI Codex. This is following my suggestion in #25, and the following benchmarks of compression, where zlib stands out as good. To combat the memory requirements though, we need to use a smaller Window Size, and we need to restructure the uzlib api a bit to support it. No API we could use, has a low state size while also offering streaming APIs, so we've adapted uzlib to get that.
od_zlib_stream_push,od_zlib_stream_poll, ..OPENDISPLAY_DECOMPRESSION_CHUNK_SIZEOPENDISPLAY_ZLIB_WINDOW_BITSTODO
ZIP, likeSTREAMING-ZIP, would signal clients that we support 512 byte window sizes.ZIP + STREAMING-ZIPmeans any window size is possible.STREAMING-ZIP, we know it supports deflate, but only at a 512 byte window size. Old clients will not understandSTREAMING-ZIP, and will assume compression is not possible (as today)windowSize=9when compressing forSTREAMING-ZIP-only devices; and to not early exit if the compressed bytes would be too large.TARGET_LARGE_MEMORY(?)Fixes #25