Verify a chart file against its container's checksum - #71
Merged
Conversation
beetlebugorg
force-pushed
the
fix/ingest-integrity
branch
from
September 5, 2026 02:23
bc02f2c to
ce04edb
Compare
beetlebugorg
force-pushed
the
fix/ingest-integrity
branch
from
September 6, 2026 19:23
ce04edb to
511aa59
Compare
beetlebugorg
force-pushed
the
fix/ingest-integrity
branch
from
September 6, 2026 20:27
511aa59 to
792c1a9
Compare
beetlebugorg
force-pushed
the
fix/ingest-integrity
branch
from
September 6, 2026 23:08
792c1a9 to
4db7dd9
Compare
beetlebugorg
force-pushed
the
fix/ingest-integrity
branch
from
September 6, 2026 23:27
4db7dd9 to
789aa8d
Compare
beetlebugorg
force-pushed
the
fix/ingest-integrity
branch
from
September 7, 2026 02:01
789aa8d to
76afb38
Compare
Both ingest formats hold a checksum for a chart file, and both checksums were discarded, so no step between the bytes on a card and the tiles a mariner navigates by verified that a chart is the one the hydrographic office published. Every zip entry has a CRC32 of its uncompressed bytes, and zip.Iterator.Entry holds it. readAlloc hashes what it read and refuses a mismatch. Raw deflate has no checksum of its own, so a flipped bit inside a literal run inflates to bytes that parse: in an SG2D field that moves a depth-area boundary vertex by up to 25 metres at COMF 1e7, and the chart bakes and draws without an error. S-57 Part 3 3.4 lets the catalogue give a CRC per file in the CATD CRCS subfield. decodeCATD read parts 0 to 6 and left CRCS unread, and CatalogEntry had no field for it. It now holds the text, catalogCrc reads the hex, and a cell opened by catalogue name is checked against it. A producer omitting it is a case the spec allows, and those cells open as before. A test flips one byte in a stored entry and reads it back.
The catalogue check covered the base cells of a directory open. Two gaps were left either side of it. The update files went unchecked on every path. A catalogue gives a CRC per file, updates included, but only .000 entries were compared, so a damaged .001 applied to a chart and the chart drew as though it were the one the hydrographic office published. The open now keeps every CRC the catalogue gives and compares each update as the chain reads it. A mismatch stops the chain and keeps the cell, the policy a corrupt update already follows, and names the file. The bake checked no file at all. It walks for .000 files and never opened the catalogue, so the two ingest paths disagreed about the same bytes: the inventory call refused a set the bake turned into archives without a word. The catalogue is now read once per exchange set and each cell and its updates are compared before the bake reads them. A file with no CRC in the catalogue passes, as before, and a set with no catalogue bakes unchanged. A damaged update fails the whole cell on the bake rather than truncating its chain. The bake writes one archive per cell, and an archive built to an earlier update than the set names cannot be told apart from a complete one afterwards. Checked against an exchange set built from a real cell, with one byte flipped: in the base, and in an update. Both paths now name the file and produce no archive, and the intact set is unaffected.
beetlebugorg
force-pushed
the
fix/ingest-integrity
branch
from
September 7, 2026 02:15
76afb38 to
5639dec
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both ingest formats hold a checksum for a chart file, and both were discarded. A flipped bit inside a deflate literal run inflates to bytes that parse: in an SG2D field it moves a depth-area boundary vertex by up to 25 metres at COMF 1e7, and the chart bakes and draws without an error. Stacked on #70.
Every zip entry has a CRC32 of its uncompressed bytes.
readAllochashes what it read and refuses a mismatch. S-57 Part 3 3.4 lets the catalogue give a CRC per file in the CATD CRCS subfield, whichdecodeCATDleft unread. Cells, their update files and the bake are all checked against it now. A producer omitting the CRC is a case the spec allows, and those files open as before.Checked on an exchange set built from a real cell with a generated catalogue, one byte flipped in the base and then in an update. Both ingest paths name the file and produce no archive.
zig build,zig build testandzig fmt --checkpass.