feat(dataset): list reported but unproved blocks in their own file - #17
Merged
Merged
Conversation
data/reported-blocks.jsonl keeps the ten blocks a contemporary source reported as invalid but whose failure the dataset cannot establish, with what is missing and the reports. CI checks their identity, any recovered header, the sources, the ordering and that none is an admitted record; it does not verify the reports. The header identity and proof-of-work check is now one helper shared with the main dataset.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ten blocks were reported as invalid by a contemporary source but cannot be admitted, because the header or body that would establish the failure is lost.
data/reported-blocks.jsonlnow keeps them in the repository, one line each with the reported height and hash, the recovered header where one exists, the claimed failure in a few words, and the reports as URLs (schema section). They are not records: nothing in the rule tables applies to them, and a body or proof file named for one is an orphan until it becomes a record.The ten are three Eligius overpayment reports from September 2012, of which 197883 has a surviving header but not its other transaction; P2Pool's 212048, known from a December 2012 IRC report joined to a node-history dump; five version-2 hashes from the March 2017 bitcoin-dev message that still lack headers; and Bitcoin Unlimited's 450529, whose header the BlockCypher capture preserves but whose reported 1,000,023-byte body no one has. The notes point at issues labelled
reported, to be opened once this merges, one per incident, holding what has already been searched.CI checks the ledger after the main dataset, through the same JSONL reader (LF endings, final newline, no duplicate keys): field shapes, that a recovered header hashes to its hash and meets its own target, that every source is an HTTP(S) URL, the ordering, and that no hash is already an admitted record. It does not verify the reports. The header identity and proof-of-work check that the main dataset already ran is lifted into one small helper so both files use it.
Testing
Offline
python ci/sanity-check.pyreports the unchanged 143 blocks, 16 block files and 85 proof files, plus 10 reported blocks not admitted; the two recovered headers hash correctly and meet their targets. All 44 tests pass: the new one runs the real ledger clean and rejects an admitted hash, a header that does not hash to its record, an empty source list, disorder and a duplicate row. Every source URL was fetched and resolves, except the gnusha archive, which refuses scripted fetches and opens in a browser; the five version reports list its mail-archive mirror as well.