fix(bdk_electrum_streaming): Refetch anchors for blocks evicted by a reorg

[evanlinjin]

Fixes #12. Fixes #19.

A reorg that moves a transaction into a different block of the same height leaves its Electrum script status untouched — the status is a hash over txid-height pairs — so the server never notifies the script hash. The anchor we already delivered goes on pointing at a block that is no longer in the chain, nothing ever asks again, and the transaction stops being canonical for good.

The fix drives the repair from the chain tip, which is the one thing that does report the eviction, and routes it through the existing SpkJob rather than a new job type: on eviction we raise the script hash notification the server will not send.

1. A completed chain job compares the checkpoint chain before and after (evicted_heights).
2. SpkHistories::spk_hashes_at_heights says which scripts had a transaction at those heights.
3. Each affected script gets SpkJob::new(&cache, spk_hash, Some(status)) — its own notification, raised locally. The status is cached with the history it stands for, so the replay costs no round-trip.
4. advance_anchors re-resolves every (height, txid) pair against whichever block the chain now has at that height. The new block is a cache.anchors miss, so the proof is refetched.

A second, independent way to end up on a chain the server has left is fixed here too: ChainJob::new replaces chain_job on every tip notification, but the replacement asks for the same heights, so its request::Headers is byte-identical and ReqQueuer deduplicates it against the request already in flight. Nothing is sent, and the answer already on its way fills the new job with the superseded chain. That is #19, and it is fixed at both layers. ReqCoord::forget_job drops the replaced job's in-flight requests, so the stale answer is ignored and the replacement's request actually goes on the wire — the target check alone would not do this, since the dedup entry only clears when the response is popped. And ChainJob now keeps the block it was created to reach and checks the assembled update against it, so a batch answered from a chain the server never announced — a reorg between receiving the request and replying to it — is rejected rather than adopted. forget_job alone would not do that either: the request is current, only the answer is not.

The last commit is separable: it folds the three co-varying spk maps into one SpkHistories with private fields, so the status can no longer drift from the history it stands for, and makes the cache serde-persistable. response::Tx is Deserialize-only upstream, so histories round-trip through a private mirror until bitcoindevkit/electrum_streaming_client#20 lands.

Breaking changes (→ 0.6.0)

• Cache loses failed_anchors. spk_histories keeps its name but changes type from HashMap<ElectrumScriptStatus, Vec<response::Tx>> to SpkHistories — a silent type change, so it breaks at construction and use, not at a field read. Cache::default() is unaffected.
• SpkJobStage::ProcessingTxsAndAnchors gains anchors_resolved — exhaustive patterns break.
• ChainJob::try_finish returns ChainJobOutcome rather than Result<CheckPoint, ChainJob>.
• ReqCoord::pop returns Option<PoppedRequest>; ReqCoord::clear is removed (no callers, and State::reset deliberately resends pending_requests()).

Additive: PoppedRequest, ReqCoord::bump_chain_generation, ReqCoord::forget_job, ChainJobOutcome, SpkHistories, State::spk_histories(), and Serialize/Deserialize on Cache and SpkHistories. New dependency serde, already in the tree via serde_json.

Testing

cargo test — 19 deterministic (tests/state.rs), 5 live (tests/env.rs), 6 unit. Each fix was reverted in turn to confirm a test catches it.

reorg_to_same_height_block_refetches_anchor_live confirms a tracked tx, then re-mines it into a different block at the same height. Fails on main, passes here.

---

Rebased onto main (2250918), and force-pushed since to mark the breaking commits, so review comments anchored to old SHAs will show as outdated. Earlier review rounds are addressed in the threads themselves.

[LLFourn]

Thanks for this — the diagnosis in #12 is right, the chain-driven design is the right
answer to it, and the write-up made the review much faster than it would otherwise have
been. I reproduced the bug against a live electrs and confirmed this branch fixes it.

Requesting changes. The first two are defects in the new code, both on the path this PR
exists to fix, and I have a failing test for each; the third is the modelling issue they
share, which I think is the one worth acting on first:

1. state.rs:256 — the stale-response guard covers the successful merkle answer but
   not the error answer. A server answers from the chain it had when it received the
   request; if a reorg has landed since, that error is about a block we have left
   behind. give_up_on_anchor blames it on the replacement block and writes two
   irreversible records — a permanent failed_anchors verdict, and the deletion of the
   txid from txids_by_height, which is the only record that would let the eviction
   path ask again. Since advance_anchors treats a failed anchor as resolved by
   omission, the job reports success with the anchor silently missing and the
   transaction stops being canonical for good. Three-line fix.

2. anchor_job.rs:55 — the refetch depends on the server answering in request order.
   A same-height reorg is applied by ChainJob's short-circuit without ever caching the
   replacement header, so the job asks for the header and the proof together. Proof
   first ⇒ cancel_jobs ⇒ anchor_job = None, unrecoverable; the header arriving
   second returns Ok(None) without advancing anything, so it cannot rescue it either.
   The protocol carries request ids precisely because ordering is not promised.

Underneath both: failed_anchors is a monotonic negative cache, which is only sound
for negatives that were proved. Right now it has two writers — a merkle root mismatch
(a proof) and any JSON-RPC error on get_merkle (not one). Keeping the connection up on
an error is right and well argued; recording a permanent disproof from it is the part I
would drop. Enforcing "only a verified proof writes failed_anchors" would make defect 1
unwriteable rather than fixed.

3. state.rs:468 — failed_anchors is a monotonic negative cache, and one of its two
   writers has not proved anything. A merkle root mismatch is a proof; a JSON-RPC error
   is not, and the predicate is the method name alone. So a rate limit, a syncing index or
   a daemon hiccup each permanently unconfirm a transaction. Worth stressing that the fix
   in (1) does not cover this — that guard closes the stale-chain case, this is a fault
   on the current chain. I ran the live unconfirming reorg against electrs to see what
   real classification would have to work with: the payload is a bare JSON string,
   "tx not found or is unconfirmed", which conflates the fault with the benign case. So
   I'd drop the durable write rather than try to classify it, and keep a classifier only
   as an optimisation.

The rest is smaller: cancel_jobs(JobId::Anchor) being terminal for the one job nothing
rebuilds; the GetHeader generation guard passing all 5 tests when deleted; the
breaking-change list missing SpkJobStage; txids_by_height retaining far more than the
~21 heights that can ever be read, which I'd like bounded. Details inline.

On testing — tests/env.rs runs fine here (cargo test --test env, 3 passed, 20s), so
the live path was never exercised in either direction. I have written two live tests:
one reproduces #12 end to end (fails on main, passes here), the other pins the
connection-survival fix (fails at 388d60a, passes here). Happy to hand both over along
with the fixes as a patch.