Skip to content

feat: collapse per-row RPC into batched reads, and add eleven datasets - #1

Merged
KonScanner merged 1 commit into
mainfrom
feat/batched-reads-and-eleven-datasets
Aug 21, 2026
Merged

KonScanner merged 1 commit into
mainfrom
feat/batched-reads-and-eleven-datasets

Conversation

@KonScanner

Copy link
Copy Markdown
Owner

Overview

Two batching routes collapse per-row RPC into per-batch RPC for blocks,
nonces, codes, slots, balances and proxy_slots, and eleven new
datasets cover the token and proxy standards that had no dataset at all.
Five silent correctness bugs found in the existing per-row paths are fixed
alongside.

Reasoning

Most datasets read one thing per row and cost one request per row. Both new
routes are on by default and both fall back to the per-row path when an
endpoint refuses, so they change what results cost and never the results
themselves.

JSON-RPC batching puts N calls in one HTTP body. It asks nothing of the
node beyond JSON-RPC itself. The batch size negotiates itself down when a
provider caps it (OP Mainnet answers 413 above ten calls, Base returns
-32014), and the discovered size is kept for the rest of the run.

State overrides replace the target's code with a 25-byte extractor loop,
turning "read one slot" into "read every slot I name". The extractor is
PUSH0-free deliberately: eth_call at a historical block runs under that
block's fork rules, and PUSH0 only became valid at Shanghai (mainnet
17034870).

The dangerous case is an endpoint that accepts the override parameter and
ignores it, because our slot keys would then reach the real contract and its
fallback output would be decoded as storage. Three defences: an exact
32*n length check, a sentinel slot set through stateDiff and read back as
the final word, and a consecutive-failure count that stops asking an endpoint
that fails the same way three times running. stateDiff is used and never
state, because state replaces an account's whole storage and would return
a well-formed file of zeros.

eth_getStorageValues sits above both rungs where a node has it — it takes a
{contract: [slots]} map, which an injected extractor cannot do at any batch
size. Support is not cached either way, because a load-balanced pool holds a
mix of node versions.

Fixes

Reading the per-row paths closely enough to batch them turned up five bugs,
all silent:

  • contracts held each hash column's value in the other column.
    code_hash is the usual join key for contract identity, so every match
    against an EXTCODEHASH found nothing rather than erroring.
  • Eleven log datasets accepted --address, --from-address, --to-address
    and --topic0..3 under --txs, printed them in the run summary, and then
    ignored them. The run returned rows it had been asked to exclude.
  • call_outcome recognised four of geth's execution-failure messages and
    treated the other ten as node failures, failing whole chunks where a revert
    wrote one null.
  • --multicall-require-success was a no-op on a batch of one — which, since
    rows are grouped by block, is every batch for a single address over a range.
  • Five calls_for_row implementations built their target with
    Address::from_slice, which panics on a width mismatch, aborting the worker
    task for a whole batch.

Tests

New coverage: 30 tests across state_override.rs, rpc_batch.rs,
errors.rs, contracts.rs and the eleven new datasets. The regression tests
for the contracts hash transposition, the non-indexed Upgraded log and the
batch-chunking property were each verified by reintroducing the bug and
confirming the test fails.

Every schema block in the new book pages was diffed byte-for-byte against
triodion help <DATASET> rather than written by hand. All eleven match.

The request-count claim was measured through a counting proxy against
mainnet: blocks -b 21000000:+500 sends 6 HTTP requests batched against 501
unbatched, and the two parquet files are byte-identical.

cargo +nightly fmt --all --check          # clean
cargo clippy --workspace --all-targets --all-features -- -D warnings   # exit 0
cargo test --workspace                    # 207 passed, 0 failed
cargo check --manifest-path crates/python/Cargo.toml                   # exit 0

crates/python is workspace-excluded, so --workspace silently skips it; it
is checked separately because this change adds three Args fields that its
adapters construct.

Most datasets read one thing per row, and cost one request per row. Two new
routes collapse that. Both are on by default, and both fall back to the
per-row path when an endpoint refuses, so batching changes what results cost
and never the results themselves.

JSON-RPC batching sends one HTTP body per N rows for blocks, nonces and
codes. On `blocks -b 21000000:+500`, 501 requests become 6, and the two
parquet files are byte-identical. The batch size negotiates itself down when
a provider caps it (OP Mainnet answers 413 above ten calls, Base returns
-32014) and the discovered size is kept for the rest of the run.

State overrides serve slots, balances and proxy_slots. `eth_call` takes a
third parameter that applies changes to a scratch copy of state, so replacing
the target's code with a 25-byte extractor loop turns "read one slot" into
"read every slot I name". One opcode swap gives that loop four readers:
SLOAD, BALANCE, EXTCODEHASH and EXTCODESIZE. The account readers take their
address from the stack, so they run at a scratch address and batch across
unrelated addresses without touching any real contract.

The extractor is PUSH0-free on purpose. `eth_call` at a historical block runs
under that block's fork rules, and PUSH0 only became valid at Shanghai
(mainnet 17034870), so Dedaub's published 23-byte form is rejected below it.

An endpoint that accepts the override parameter and ignores it is the
dangerous case: our slot keys would reach the real contract, and its fallback
output would be decoded as storage. Two checks prevent that. The return must
be exactly 32*n bytes, and the storage reader sets a sentinel slot through
stateDiff and reads it back as the final word. stateDiff is used and never
state, because state replaces an account's whole storage and would return a
well-formed file of zeros.

Each reader has one blind spot: the input for which our own override is the
answer. It is the scratch address for the account readers and the sentinel
slot for the storage reader. Both would pass the length check and the
sentinel check, so neither is detectable downstream. A row that names either
one skips the batch and is read with no override in sight.

An endpoint that fails the same way three times in a row stops being asked.
Without that count, a proxy that strips the override below the JSON-RPC layer
takes the full halving cascade on every chunk, for the length of the run.

eth_getStorageValues sits above both rungs where a node has it. It takes a
{contract: [slots]} map, so slots for different contracts at one block travel
in a single request, which an injected extractor cannot do at any batch size.
Support cannot be cached either way, because a load-balanced pool holds a mix
of node versions, so consecutive misses are counted rather than a verdict
latched.

address_appearances made three RPCs in series per block and now makes them
concurrently.

Eleven datasets are new: approvals_for_all, contract_interfaces,
erc1155_metadata, erc1155_transfers, erc20_allowances, erc2612_nonces,
erc4626_metadata, erc4626_vault_events, erc777_transfers, proxy_slots and
proxy_upgrades. erc20_allowances reads allowance() at a block and sits beside
the existing erc20_approvals, which replays the event log. An allowance also
falls when the spender spends, because transferFrom decrements it without an
Approval event, so the log alone overstates what is spendable.

proxy_upgrades takes the implementation address from topic1 when it is
indexed and from the data word when it is not. `indexed` is not part of an
event signature preimage, so zos-lib's 2018 `Upgraded(address)` and the
ERC-1967 one share a topic0. Selecting on the argument's position rather than
its presence fetches those logs and then discards them, which would report
USDC's proxy as never upgraded.

New flags: --no-batch-rpc-calls, --no-batch-state-reads and
--state-override-batch-size.

Reading the per-row paths closely enough to batch them turned up five
correctness bugs, all of them silent.

contracts held each hash column's value in the other column: init_code_hash
carried keccak(code) and code_hash carried keccak(init_code). code_hash is
the usual join key for contract identity, so every match against an
EXTCODEHASH, or against an external codehash index, found nothing rather than
erroring.

Eleven log datasets accepted --address, --from-address, --to-address and
--topic0..3 under --txs, counted them into the partition set, printed them in
the run summary, and then ignored them. A receipt query asks for one whole
transaction, so the narrowing that the by-block path pins into eth_getLogs
has to be re-applied by hand, and was not. The run returned rows it had been
asked to exclude and said nothing.

call_outcome recognised four of geth's execution-failure messages and treated
the other ten as node failures, so a stack underflow or a "return data out of
bounds" failed a whole chunk where a revert wrote one null. EIP-1474 reserves
error code 3 for execution reverted, which is the one structured signal
available, and it is now checked before the message. contract_interfaces was
the most exposed, because it aims eth_call at arbitrary bytecode.

--multicall-require-success was a no-op on a batch of one. Every dataset
groups its rows by block before batching, so one address over a block range
gives one row per group and every batch is a singleton: the flag asked for
strictness and got the default nulls.

Five multicall calls_for_row implementations built their target with
Address::from_slice, which panics on a width mismatch, and --address is
hex-decoded with no length check. A panic there aborts the worker task for a
whole batch of rows, where an error demotes the batch and reports one.

--l1-rpc's help named six datasets that read it. None of them exist; nothing
reads it yet. It connects and reports its chain id, and the help now says so.

Documentation covers the eleven datasets, the two batching routes and the
three flags. Every schema block in those pages is copied from `triodion help
<DATASET>` rather than derived by hand.
@KonScanner
KonScanner merged commit 34d6e4f into main Aug 21, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant