Repository navigation
Raise zip entry cap to 8 GiB, configurable via env - #3
Merged
Merged
Conversation
au/vic/statewide ships a 3.2 GB parcels zip whose VMPROP.gdb/a00000014.gdbtable declares 2,540,059,567 bytes, over the 2 GiB per-entry cap, so the source failed. The cap now defaults to 8 GiB and BATCH_MACHINE_MAX_ZIP_ENTRY_BYTES overrides it. Claude-Session: https://claude.ai/code/session_01VHQ8tPQxvwtMYJfu4EGeos
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
au/vic/statewideparcels (3.2 GB zip) containsVMPROP.gdb/a00000014.gdbtable, which declares 2,540,059,567 bytes. The zip-bomb guard from c90ee0e (MAX_ZIP_ENTRY_BYTES = 2 GiB) rejects it, so the source has failed in RunBatchMachineParcels since 2026-08-28.Change
BATCH_MACHINE_MAX_ZIP_ENTRY_BYTES(integer bytes) overrides it, read on each extraction.Disk space
Extraction goes to
<workdir>/Unzipped/(ZipDecompressTask.decompress), where workdir is anmkdtemp(prefix='conform-', dir=destdir)directory inopenaddr/__init__.py. The cached zip is also copied into that workdir, so a job needs roughly the archive size plus the full extracted size on that volume. There is no free-space check anywhere in the repo, so an exhausted disk fails mid-extraction. The 8 GiB cap bounds a single entry only, not the sum of entries. Worker temp space for this source should be at least ~10 GB; I did not add a disk check.Tests
Added to
TestZipDecompressTask: entry under the cap extracts, default is 8 GiB and admits the 2.54 GB entry, env override refuses an oversized entry, env override raises a lowered cap. Existing oversized-entry test still passes.python3 -m unittest openaddr.tests.conform.TestZipDecompressTask(10 tests OK) andpython3 test.py(128 tests OK) run in the Docker image. Notetest.pydoes not importTestZipDecompressTask, so CI does not run these tests today.After merging
The amd64 image must be rebuilt and pushed (
registry.geocod.io/geocodio/batch-machine:latest), and the workers mustdocker pullit. The registry proxy caches:latest, so a manual pull is needed for workers to see the new image.https://claude.ai/code/session_01VHQ8tPQxvwtMYJfu4EGeos