Skip to content

Add full bodies for 8 stale blocks recovered from public btcd peers - #158

Merged
0xB10C merged 1 commit into
bitcoin-data:masterfrom
john-a-zoidburg:data/recover-btcd-stale-blocks
Oct 10, 2026
Merged

0xB10C merged 1 commit into
bitcoin-data:masterfrom
john-a-zoidburg:data/recover-btcd-stale-blocks

Conversation

@john-a-zoidburg

Copy link
Copy Markdown
Contributor

Summary

Adds complete serialized bodies for eight existing stale-block headers, spanning June 2016 to April 2026: 10,423,119 bytes and 16,614 transactions, including coinbases and applicable witness data.

The change adds eight .bin files under blocks/. Each starts with the byte-identical header already recorded in stale-blocks.csv; the CSV is unchanged.

Provenance and recovery

The bodies were downloaded on September 26, 2026 UTC from public Bitcoin P2P endpoints identifying themselves as btcd peers. Peers were discovered through saved captures of the BTCNodes public snapshot API. Standard getdata requests for the already-known hashes returned the full blocks, using MSG_WITNESS_BLOCK where supported. The contributed files preserve those response bytes unchanged.

btcd was a useful recovery target because its block-serving path fetches stored blocks by hash without Bitcoin Core's stale-block age restriction. Core only serves blocks outside its active chain if they have passed script validation and are within 30 days of its best header, measured by both timestamp and proof-equivalent work. This restriction applies even when Core still has the older data on disk. btcd can therefore serve much older stale blocks when it still retains them, as these recoveries demonstrate.

Existing header Header date, UTC Bytes Transactions
414283 2016-06-01 998,119 1,562
415144 2016-06-07 481,406 1,089
488710 2017-10-07 1,038,101 1,439
497373 2017-12-03 1,037,052 2,263
656641 2020-11-12 1,309,539 2,477
921987 2025-11-03 1,849,109 4,290
936988 2026-02-16 1,974,437 1,504
946239 2026-04-22 1,735,356 1,990

Validation

  • Independently checked complete serialization, header identity and proof of work, coinbase placement, transaction uniqueness, transaction Merkle roots and applicable witness commitments. All eight build directly on canonical parents. Neither they nor their ancestry match the current proved-invalid catalogue or its reported-invalid entries.
  • The complete modified repository passes the unchanged ci/sanity-check.py, ci/check-with-bitcoind.py and git diff --check. The Core check used Bitcoin Core 31.1 with P2P disabled: 1,395 block files checked without rejection. Six pre-existing headers have unknown parents; all eight contributed blocks were checked.
  • Additional offline verification used Bitcoin Core consensus functions at commit 05e49b3, checking 48,566 input scripts and the applicable historical header, activation, maturity, finality, relative-lock, value, subsidy, fee, weight and sigop rules. Funding transactions were authenticated with Merkle proofs against canonical headers. Required external input availability was supported by authenticated canonical spends at the stale-block height or later, with the one exception described below. Native negative controls failed as expected.

Validation scope: the additional checks reconstructed the required input state; they did not replay the full historical chainstate. For one input in 946239, availability was corroborated by current unspent observations from a local Esplora index and Blockstream Esplora, rather than an authenticated canonical spend.

Checks ran offline or on isolated instances with P2P disabled. No block or transaction was broadcast.

Transaction observations

The recovered bodies contain 12 ordinary transactions absent from the canonical-chain indexes checked during analysis: one in 497373, one in 921987 and ten in 946239. Ten directly conflict with confirmed transactions; the other two depend on those conflicting transactions.

Conversely, one transaction in 488710 confirmed 2,283 blocks later, at height 490993. Its exact bytes were verified in the later canonical block, illustrating why a short lookahead can misclassify a transaction as absent from the surviving chain.

Fill existing header-only entries with original serialized block data
downloaded over Bitcoin P2P on 2026-09-26, preserving witness data.
@deadmanoz

Copy link
Copy Markdown
Contributor

Brilliant @john-a-zoidburg, thanks again!

@deadmanoz

Copy link
Copy Markdown
Contributor

Does the BTCNodes public snapshot API contain most btcd peers? As in, is it worth concerted effort to query btcd nodes for other blocks?

@john-a-zoidburg

Copy link
Copy Markdown
Contributor Author

I tried all of the publicly connectable btcd nodes that I could easily find, however there could definitely be more.

The bigger win might come from nodes that do not have open inbound connections, and only make their own outbound connections. I was thinking we could set up a custom modified bitcoin core build which would sit and wait for btcd nodes to connect to it, and then request known state blocks.

@deadmanoz

Copy link
Copy Markdown
Contributor

I tried all of the publicly connectable btcd nodes that I could easily find, however there could definitely be more.

The bigger win might come from nodes that do not have open inbound connections, and only make their own outbound connections. I was thinking we could set up a custom modified bitcoin core build which would sit and wait for btcd nodes to connect to it, and then request known state blocks.

Yeah, clever idea!

@0xB10C

0xB10C commented Oct 1, 2026

Copy link
Copy Markdown
Collaborator

There's the getblockfrompeer RPC which you could use. However, I'm not sure how this behaves with 1) requesting many blocks, 2) potentially not receiving these blocks (I don't remember if we disconnect after a timeout; Your LLM can probably figure it out).

@john-a-zoidburg

john-a-zoidburg commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor Author

Prototype P2P network stale block collector - john-a-zoidburg/bitcoin#1

@0xB10C

0xB10C commented Oct 10, 2026

Copy link
Copy Markdown
Collaborator

Looks good to me! Thanks again for all these.

@0xB10C
0xB10C merged commit 352edf9 into bitcoin-data:master Oct 10, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants