Repository navigation
feat: collapse per-row RPC into batched reads, and add eleven datasets - #1
Merged
Merged
Conversation
Most datasets read one thing per row, and cost one request per row. Two new
routes collapse that. Both are on by default, and both fall back to the
per-row path when an endpoint refuses, so batching changes what results cost
and never the results themselves.
JSON-RPC batching sends one HTTP body per N rows for blocks, nonces and
codes. On `blocks -b 21000000:+500`, 501 requests become 6, and the two
parquet files are byte-identical. The batch size negotiates itself down when
a provider caps it (OP Mainnet answers 413 above ten calls, Base returns
-32014) and the discovered size is kept for the rest of the run.
State overrides serve slots, balances and proxy_slots. `eth_call` takes a
third parameter that applies changes to a scratch copy of state, so replacing
the target's code with a 25-byte extractor loop turns "read one slot" into
"read every slot I name". One opcode swap gives that loop four readers:
SLOAD, BALANCE, EXTCODEHASH and EXTCODESIZE. The account readers take their
address from the stack, so they run at a scratch address and batch across
unrelated addresses without touching any real contract.
The extractor is PUSH0-free on purpose. `eth_call` at a historical block runs
under that block's fork rules, and PUSH0 only became valid at Shanghai
(mainnet 17034870), so Dedaub's published 23-byte form is rejected below it.
An endpoint that accepts the override parameter and ignores it is the
dangerous case: our slot keys would reach the real contract, and its fallback
output would be decoded as storage. Two checks prevent that. The return must
be exactly 32*n bytes, and the storage reader sets a sentinel slot through
stateDiff and reads it back as the final word. stateDiff is used and never
state, because state replaces an account's whole storage and would return a
well-formed file of zeros.
Each reader has one blind spot: the input for which our own override is the
answer. It is the scratch address for the account readers and the sentinel
slot for the storage reader. Both would pass the length check and the
sentinel check, so neither is detectable downstream. A row that names either
one skips the batch and is read with no override in sight.
An endpoint that fails the same way three times in a row stops being asked.
Without that count, a proxy that strips the override below the JSON-RPC layer
takes the full halving cascade on every chunk, for the length of the run.
eth_getStorageValues sits above both rungs where a node has it. It takes a
{contract: [slots]} map, so slots for different contracts at one block travel
in a single request, which an injected extractor cannot do at any batch size.
Support cannot be cached either way, because a load-balanced pool holds a mix
of node versions, so consecutive misses are counted rather than a verdict
latched.
address_appearances made three RPCs in series per block and now makes them
concurrently.
Eleven datasets are new: approvals_for_all, contract_interfaces,
erc1155_metadata, erc1155_transfers, erc20_allowances, erc2612_nonces,
erc4626_metadata, erc4626_vault_events, erc777_transfers, proxy_slots and
proxy_upgrades. erc20_allowances reads allowance() at a block and sits beside
the existing erc20_approvals, which replays the event log. An allowance also
falls when the spender spends, because transferFrom decrements it without an
Approval event, so the log alone overstates what is spendable.
proxy_upgrades takes the implementation address from topic1 when it is
indexed and from the data word when it is not. `indexed` is not part of an
event signature preimage, so zos-lib's 2018 `Upgraded(address)` and the
ERC-1967 one share a topic0. Selecting on the argument's position rather than
its presence fetches those logs and then discards them, which would report
USDC's proxy as never upgraded.
New flags: --no-batch-rpc-calls, --no-batch-state-reads and
--state-override-batch-size.
Reading the per-row paths closely enough to batch them turned up five
correctness bugs, all of them silent.
contracts held each hash column's value in the other column: init_code_hash
carried keccak(code) and code_hash carried keccak(init_code). code_hash is
the usual join key for contract identity, so every match against an
EXTCODEHASH, or against an external codehash index, found nothing rather than
erroring.
Eleven log datasets accepted --address, --from-address, --to-address and
--topic0..3 under --txs, counted them into the partition set, printed them in
the run summary, and then ignored them. A receipt query asks for one whole
transaction, so the narrowing that the by-block path pins into eth_getLogs
has to be re-applied by hand, and was not. The run returned rows it had been
asked to exclude and said nothing.
call_outcome recognised four of geth's execution-failure messages and treated
the other ten as node failures, so a stack underflow or a "return data out of
bounds" failed a whole chunk where a revert wrote one null. EIP-1474 reserves
error code 3 for execution reverted, which is the one structured signal
available, and it is now checked before the message. contract_interfaces was
the most exposed, because it aims eth_call at arbitrary bytecode.
--multicall-require-success was a no-op on a batch of one. Every dataset
groups its rows by block before batching, so one address over a block range
gives one row per group and every batch is a singleton: the flag asked for
strictness and got the default nulls.
Five multicall calls_for_row implementations built their target with
Address::from_slice, which panics on a width mismatch, and --address is
hex-decoded with no length check. A panic there aborts the worker task for a
whole batch of rows, where an error demotes the batch and reports one.
--l1-rpc's help named six datasets that read it. None of them exist; nothing
reads it yet. It connects and reports its chain id, and the help now says so.
Documentation covers the eleven datasets, the two batching routes and the
three flags. Every schema block in those pages is copied from `triodion help
<DATASET>` rather than derived by hand.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Two batching routes collapse per-row RPC into per-batch RPC for
blocks,nonces,codes,slots,balancesandproxy_slots, and eleven newdatasets cover the token and proxy standards that had no dataset at all.
Five silent correctness bugs found in the existing per-row paths are fixed
alongside.
Reasoning
Most datasets read one thing per row and cost one request per row. Both new
routes are on by default and both fall back to the per-row path when an
endpoint refuses, so they change what results cost and never the results
themselves.
JSON-RPC batching puts N calls in one HTTP body. It asks nothing of the
node beyond JSON-RPC itself. The batch size negotiates itself down when a
provider caps it (OP Mainnet answers
413above ten calls, Base returns-32014), and the discovered size is kept for the rest of the run.State overrides replace the target's code with a 25-byte extractor loop,
turning "read one slot" into "read every slot I name". The extractor is
PUSH0-free deliberately:
eth_callat a historical block runs under thatblock's fork rules, and PUSH0 only became valid at Shanghai (mainnet
17034870).
The dangerous case is an endpoint that accepts the override parameter and
ignores it, because our slot keys would then reach the real contract and its
fallback output would be decoded as storage. Three defences: an exact
32*nlength check, a sentinel slot set throughstateDiffand read back asthe final word, and a consecutive-failure count that stops asking an endpoint
that fails the same way three times running.
stateDiffis used and neverstate, becausestatereplaces an account's whole storage and would returna well-formed file of zeros.
eth_getStorageValuessits above both rungs where a node has it — it takes a{contract: [slots]}map, which an injected extractor cannot do at any batchsize. Support is not cached either way, because a load-balanced pool holds a
mix of node versions.
Fixes
Reading the per-row paths closely enough to batch them turned up five bugs,
all silent:
contractsheld each hash column's value in the other column.code_hashis the usual join key for contract identity, so every matchagainst an
EXTCODEHASHfound nothing rather than erroring.--address,--from-address,--to-addressand
--topic0..3under--txs, printed them in the run summary, and thenignored them. The run returned rows it had been asked to exclude.
call_outcomerecognised four of geth's execution-failure messages andtreated the other ten as node failures, failing whole chunks where a revert
wrote one null.
--multicall-require-successwas a no-op on a batch of one — which, sincerows are grouped by block, is every batch for a single address over a range.
calls_for_rowimplementations built their target withAddress::from_slice, which panics on a width mismatch, aborting the workertask for a whole batch.
Tests
New coverage: 30 tests across
state_override.rs,rpc_batch.rs,errors.rs,contracts.rsand the eleven new datasets. The regression testsfor the
contractshash transposition, the non-indexedUpgradedlog and thebatch-chunking property were each verified by reintroducing the bug and
confirming the test fails.
Every schema block in the new book pages was diffed byte-for-byte against
triodion help <DATASET>rather than written by hand. All eleven match.The request-count claim was measured through a counting proxy against
mainnet:
blocks -b 21000000:+500sends 6 HTTP requests batched against 501unbatched, and the two parquet files are byte-identical.
crates/pythonis workspace-excluded, so--workspacesilently skips it; itis checked separately because this change adds three
Argsfields that itsadapters construct.