Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .cargo-crap.toml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ allow = [
"NodeConfig::apply_kv",
"StrongTxTable::count_ones_bits",
"StrongTxTable::set_bits_range",
"TxTable::open_inwit",
"TxTable::open_seqsigwit",
"TxTable::put_spend_batch_by_abs_meta",
"UringSession::submit_and_wait_one",
"cli_main",
Expand Down
48 changes: 27 additions & 21 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,11 +103,17 @@ before 1.0).
- **Lightning chain backends (`Q-69` Open):** [`docs/lightning.md`](docs/lightning.md)
owns CLN `bcli` and ldk-node Esplora/Electrum.

- **Schema 25 `txstat.body`:** 8 B/create ULEB confirm-time econ
(`n_in`/`fee_sat`/`base`/`wit_extra`) with per-header remaining-byte overflow.
Occupied 24 open rewrites `meta` and zero-extends the file to loc count (no
`txout.body` rewrite). Unreleased leftover `txfixed.body` is unlinked. A 24
binary refuses 25 `meta`.
- **Schema 25 econ stems:** `txstat.body` stays 8 B/create: three ULEBs
(`fee_sat`/`base`/`wit_extra`) plus per-header remaining-byte overflow.
`n_in` is `input.loc` (u16). `input.body` is the parent edge (create fk
and vout) per input. `seqsigwit` is the old `inwit` stem (sequence,
scriptSig, witness); open renames those files. `txstat.*` and `input.*`
sit next to `seqsigwit` (cold when split). Open with no `input.loc`
backfills the edges from `seqsigwit` prevouts. A four-ULEB cell that
started with `n_in` is not rewritten; resync that datadir. Occupied 24
open rewrites `meta` and zero-extends `txstat.body` (no `txout.body`
rewrite). Unreleased leftover `txfixed.body` is unlinked. A 24 binary
refuses 25 `meta`.

- **Core functional `mempool_packages.py`:** inventory `run`. Verbose mempool
`vsize` / ancestor-descendant size use Core ceil-vsize; `wtxid` is on both
Expand Down Expand Up @@ -228,7 +234,7 @@ musl (operator) + Windows CRT-static PE + Darwin aarch64.
- **Faster IBD:** loc rides the InFlight pin (load never loc-by-fk); body-queue
`header_fk` skips header ensure; tip+1 hole racing by drain time; sequential
`header.body` size/weight; BIP30 same-txid across a load wave.
- **Smaller store:** schema 22 `create.loc` + `inwit.loc` (no three Class A
- **Smaller store:** schema 22 `create.loc` + `seqsigwit.loc` (no three Class A
`*.idx`); compact txout amounts; sealed fuse8 + SH occupancy mmap (heap
`fuse8=` / `mphf_occ=` only).
- **Mempool persist:** packed schema 2 — 5 s dirty tail, one-record DEAD
Expand Down Expand Up @@ -975,7 +981,7 @@ musl (operator) + Windows CRT-static PE + Darwin aarch64.
`ibd_skips_dead_peer` run in `cargo test --workspace` and `coverage.sh`.
The separate `multinode` job is gone. [`TESTING.md`](TESTING.md).

- **No loc-count echo:** packed decode already walks loc `n_out` / inwit
- **No loc-count echo:** packed decode already walks loc `n_out` / seqsigwit
`input_count`; drop the tautological len==count Corrupt. `spent_record_len(0)`
is `0×8` (no special case). Delete unused no-op `head_reserve_additional`.

Expand All @@ -984,7 +990,7 @@ musl (operator) + Windows CRT-static PE + Darwin aarch64.
`two_node_header_and_block_sync` runs under `coverage.sh`.
[`TESTING.md`](TESTING.md).

- **Schema 22:** `create.loc` + `inwit.loc` (no Class A `{txout,spent,inwit}.idx`).
- **Schema 22:** `create.loc` + `seqsigwit.loc` (no Class A `{txout,spent,seqsigwit}.idx`).
LAYOUT17 omits `output_count` (decode `n_out` from loc). Spent slot is flags +
u40 spend fk + u16 vin. `txout` amount is flags bits 4–7 = decimal
exponent (0–9) + ULEB mantissa (`sats = mantissa × 10^e`; canonical compact,
Expand Down Expand Up @@ -1533,7 +1539,7 @@ musl (operator) + Windows CRT-static PE + Darwin aarch64.
`notfound`, serve an announced tx from the tip block).

- **IBD `getdata` serve reconstructs from Class A spans:** contiguous
`header_txs` loads `txout.body` + `inwit.body` as libc sequential preads
`header_txs` loads `txout.body` + `seqsigwit.body` as libc sequential preads
(not per-tx `get_tx_full`), off the session reactor. Host probe:
`scripts/ibd-serve-bench.py`.

Expand Down Expand Up @@ -1741,7 +1747,7 @@ musl (operator) + Windows CRT-static PE + Darwin aarch64.

### Changed

- **IBD `getdata` span IO:** `txout.body` and `inwit.body` span preads run
- **IBD `getdata` span IO:** `txout.body` and `seqsigwit.body` span preads run
in parallel (SSD one volume, or `--datadir-cold` split). BIP324
`write_v2_contents` takes owned bytes (no extra `to_vec` on the serve writer).

Expand Down Expand Up @@ -2379,12 +2385,12 @@ nightly job (not a required PR check). P2P DoS is not Core-parity.
or incomplete row. See [`docs/core-functional.md`](docs/core-functional.md).
No Core scripts run in default `cargo test` yet.

- **`--datadir-cold PATH`:** Class A `inwit.body` / `inwit.idx/` (cold; ~486 GiB
- **`--datadir-cold PATH`:** Class A `seqsigwit.body` / `seqsigwit.idx/` (cold; ~486 GiB
on mainnet) live under `{PATH}/store` when set. `--datadir` still holds every
other file (`txout`, `spent`, heads, mempool, peers, cookie). Omit the flag
and both hot and cold files stay in `--datadir`. Conf: `datadir-cold=`.
Existing split: move `inwit.body` + `inwit.idx/` yourself; the hot store
records `inwit.reloc` so a later open without the flag refuses.
Existing split: move `seqsigwit.body` + `seqsigwit.idx/` yourself; the hot store
records `seqsigwit.reloc` so a later open without the flag refuses.

- **CI musl artifacts:** after a green `ci` run on `master`/`main`, workflow
`musl` builds `nix build .#rbitcoin-musl` and uploads
Expand Down Expand Up @@ -2528,7 +2534,7 @@ nightly job (not a required PR check). P2P DoS is not Core-parity.
graph wtxids.

- **`--sptweaks` backfill:** one-core completion machine (`txout`, then
`inwit`/parents only for P2TR) and batched height-blob + idx writes.
`seqsigwit`/parents only for P2TR) and batched height-blob + idx writes.
Mainnet origin→tip on SSD is typically **about 1–2 hours** (was several
hours at ~15–25 h/s serial `get_tx_full`). INFO every 10 s:
`sptweaks: backfill next=… tip=… rate=…/s remain=…`.
Expand Down Expand Up @@ -2781,7 +2787,7 @@ nightly job (not a required PR check). P2P DoS is not Core-parity.
open — wipe `store/scripthash*` and restart with `--shindex`.

- **`getblockchaininfo` disk / progress:** `size_on_disk` is a walk of
store file lengths (plus cold inwit when split). `verificationprogress`
store file lengths (plus cold seqsigwit when split). `verificationprogress`
is `blocks / headers` (1.0 when headers is 0), not a dummy 0.5 / 1.0.

- **No soak program.** Signet-first remains ordinary run advice. Q-35 is
Expand Down Expand Up @@ -3001,8 +3007,8 @@ nightly job (not a required PR check). P2P DoS is not Core-parity.
sleep in the same `select!` was reset by every perf/RPC tick, so a node
that lost its last follow peer (mainnet 962723) never redialed.

- **Class A idx rolls:** each stem (`txout` / `inwit` / `spent`) rolls its
own idx at the soft span. Inwit no longer forces hot idx splits.
- **Class A idx rolls:** each stem (`txout` / `seqsigwit` / `spent`) rolls its
own idx at the soft span. SeqSigWit no longer forces hot idx splits.
- **`strong_tx`:** always L2 (1 bit/fk). `RBITCOIN_CLASS_C_INRAM_MAX_MB`
still caps `confirmed` / `header_txs_*` only.
- **Schema 17 freeze note:** [`SCHEMA.md`](SCHEMA.md)
Expand Down Expand Up @@ -3073,12 +3079,12 @@ nightly job (not a required PR check). P2P DoS is not Core-parity.
store that already has Class A creates (schema 15/16 16-byte meta /
9-byte spent) or leftover `key_len=32` SH runs is refused. Empty
Class A still soft-opens. This is meant to be the last full-datadir
reindex for the Class A / B / C layout; later work (inwit Δfk, a new
reindex for the Class A / B / C layout; later work (seqsigwit Δfk, a new
consensus script kind) would be schema 18 and should not require
another wipe of `txout` / `spent` / heads. Layout in 17: SH runs
unique `(scripthash, create_fk)` at `key_len=40`; megakey pages are
uleb fk0+deltas; thin LAYOUT17 `txout` meta; script kinds 0–9; 8-byte
spent slots; overflow is `spent.ovf`; reserved inwit bits 4–7 and
spent slots; overflow is `spent.ovf`; reserved seqsigwit bits 4–7 and
spent flags other than `MULTI_SPENDER` are Corrupt. Leftover
`archive_epoch`, `store/wire`, and single-file `sp_tweaks.idx` /
`sp_tweaks.body` are unlinked on open. Tweaks (when `--sptweaks`) are
Expand All @@ -3098,7 +3104,7 @@ nightly job (not a required PR check). P2P DoS is not Core-parity.
lookup wave wall is `lookup_thr wave=`.

- **Confirm write path:** Class C `strong_tx` flush already wrote only the dirty
suffix — now pinned. Class A `txout`/`inwit`/`spent` bodies submit as one
suffix — now pinned. Class A `txout`/`seqsigwit`/`spent` bodies submit as one
`pwrite_batch` wave. `tx.head` insert is write-behind (page-grouped drain
overlaps structural/Class C); resolve hits a pending txid→fk map until drain.
Crash-open backfills a lagging head from Class A.
Expand Down Expand Up @@ -3144,7 +3150,7 @@ nightly job (not a required PR check). P2P DoS is not Core-parity.
Tip write-through only when `height == next_height`. Restart resumes
from `next_height`.

- **Schema 15 Class A split:** `txout.body` (outs) + `inwit.body` (ins+witness)
- **Schema 15 Class A split:** `txout.body` (outs) + `seqsigwit.body` (ins+witness)
+ `spent.body` (9 B×n_out). Packed `tx.body` with creates is refused. Pin/SH
read outs only; annotate RMW is `spent_off+9×vout`. Working-set census in
[`SCHEMA.md`](./SCHEMA.md).
Expand Down
4 changes: 2 additions & 2 deletions COMPAT.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ Full `/tx/:txid` JSON still has `vin[]`. Electrum has no outspend-vin surface.
| WTx inventory | BIP339 when peer also sends `wtxidrelay` | BIP339 |
| GetAddr | Core `MAX_ADDR_TO_SEND` / `MAX_PCT_ADDR_TO_SEND` (**1000** / **23%**), 24h per-bind cache. Named copies in `rbitcoin-net` — do not “improve” without a named reason to diverge. `MAX_ADDR_MAN` (8192) is **our** HashMap DoS cap and must stay above `1000/0.23` | Core new/tried buckets (~80k); same 1000 / 23% |
| Package submit | RPC `submitpackage` / Esplora `POST /txs/package` (no P2P package command) | BIP331 wire |
| Pruning / GUI | Unpruned: `inwit.body`. `--prune-inwit`: watermark + `NETWORK_LIMITED`, kept witness is 288 height files under `store/inwit.window/` plus a RAM cap (`--prune-inwit-ram-threshold-bytes`, `0` = files only). Not a rolling stem. Not Core `-prune` of headers/txout | Supported |
| Pruning / GUI | Unpruned: `seqsigwit.body`. `--prune-seqsigwit`: watermark + `NETWORK_LIMITED`, kept witness is 288 height files under `store/seqsigwit.window/` plus a RAM cap (`--prune-seqsigwit-ram-threshold-bytes`, `0` = files only). Not a rolling stem. Not Core `-prune` of headers/txout | Supported |
| Mining template RPC | `getblocktemplate` / `getmininginfo` / `prioritisetransaction` (selector; no stratum) | GBT + stratum / pool stack |
| Wallets | Electrum clients (requires `--shindex`) | Descriptor + legacy |
| Scripthash index | Optional (`--shindex`, default **off**); bulk at tip when on | External ElectrumX / Fulcrum; Core `-txindex` is different (txid→block) |
Expand Down Expand Up @@ -223,7 +223,7 @@ App `ServeLimits` always on (same model as Electrum).
| Tip | done | `/blocks/tip/height`, `/blocks/tip/hash`. REST stamps `X-Bitcoin-Chain-Tip` / `X-Bitcoin-Chain-Tip-Height` (CORS-exposed): **live tip** for block/tx/header routes; **SH watermark** for `/address/` and `/scripthash/` so wallet JSON matches the SH join. Empty chain omits them (existing 503). If the pin dies mid-request: **503** `chain view moved`. |
| Blocks list | done | `/blocks`, `/blocks/:start_height` (10 summaries, newest-first) |
| Block | done | `/block/:hash` JSON, `/raw`, `/status`, `/header`, `/txids`, `/txid/:i`, `/txs[/:start]`. JSON `bits` is the compact-target **u32** (Esplora schema, not Core hex). `size` / `weight` are BIP144 total size and BIP141 weight (witness included). |
| Tx | done | `/tx/:txid` full JSON, `/hex`, `/raw`, `/status`, Electrum `/merkle-proof`, BIP37 `/merkleblock-proof`, `/outspend(s)` (`vin` from the spent slot; unspent omits it). Below inwit prune: JSON is 200 partial (`pruned: true`, vout+status, empty `vin`); stamped `txstat` fills `fee`/`size`/`weight`. Mempool-only txs (not in Class A) use the wire body from the mempool hub (`vin`/`vout`/`size`/`weight`/`fee`/`sigops`, `status.confirmed` false) including `GET /tx/:txid/status`. Live `/outspend(s)` overlay mempool spends of confirmed coins; `?asof=` omits mempool. `?asof=<hash>` on `/status` and `/outspend(s)`: confirmed/spent as of that ancestor; 404 if not on chain. |
| Tx | done | `/tx/:txid` full JSON, `/hex`, `/raw`, `/status`, Electrum `/merkle-proof`, BIP37 `/merkleblock-proof`, `/outspend(s)` (`vin` from the spent slot; unspent omits it). Below seqsigwit prune: JSON is 200 partial (`pruned: true`, vout+status, empty `vin`); stamped `txstat` fills `fee`/`size`/`weight`. Mempool-only txs (not in Class A) use the wire body from the mempool hub (`vin`/`vout`/`size`/`weight`/`fee`/`sigops`, `status.confirmed` false) including `GET /tx/:txid/status`. Live `/outspend(s)` overlay mempool spends of confirmed coins; `?asof=` omits mempool. `?asof=<hash>` on `/status` and `/outspend(s)`: confirmed/spent as of that ancestor; 404 if not on chain. |
| Address / scripthash | done | stats + `/utxo` + `/txs` + `/txs/mempool` + `/txs/chain[/:last_seen_txid]` + `/txs/summary[/:last_seen_txid]` (dialect; next row). `/utxo` matches Electrum listunspent (mempool funding + drop mempool-spent confirmed); `/txs` and `/txs/mempool` use full Esplora tx JSON for mempool-only rows (wire from the hub). `?after_txid=` on `/txs` and `/txs/summary` skips through that tx (mempool then chain); unknown or unparseable → **422** `after_txid not found`. `POST /addresses/txs` and `POST /scripthashes/txs` (and `/txs/summary`) take a JSON array (max **300**; over → **422** `body too long`); unique scripts join once per request, merge newest-first. Unix/loopback `X-Rbitcoin-Client`: last-1 GET join + last-bulk POST (16 MiB packed/client), 30s idle, 256 clients, singleflight on the same `(client, sh)`. Public TCP ignores the header (stack-local join). Concurrent different GET scripts replace last-1. Unbounded process LRU stays **X-M3**. Needs SH finalize. Stamp is visible SH (durable + pending write-behind), matching live tip while jobs sit in RAM. `?asof=<hash>` on `/`, `/utxo`, `/txs`, `/txs/chain`, `/txs/summary`: confirmed join at that ancestor **at or behind visible SH**, **no** mempool; headers are the asof hash; 404 if not on chain or ahead of visible SH. |
| `/txs/summary` | dialect | **Not** in Blockstream Esplora [`API.md`](https://github.com/Blockstream/esplora/blob/master/API.md). Compact `{txid, value, height, time}` like mempool.space `/address/:addr/txs/summary`. Confirmed only (25/page, newest first); path cursor `/:last_seen_txid` and mempool.space `?after_txid=` (unknown → **422**). `value` is net sats for that script in that tx (funded − spent). `time` is the confirming header timestamp (`0` if the header is missing). Mempool rows stay on `/txs` and `/txs/mempool`. Over `--max-sh-creates` → **503**. |
| Mempool / fees | done | `/mempool`, `/mempool/txids`, `/mempool/txids/page[/:last]`, `/mempool/recent` (accept-order ring), `/fee-estimates`, mempool.space `/fees/recommended` and `/v1/fees/recommended` (sat/vB tiers) |
Expand Down
Loading
Loading