feat(cketh): sweeper fee-funding task, with an end-to-end test - #11086
feat(cketh): sweeper fee-funding task, with an end-to-end test#11086mbjorkqvist wants to merge 5 commits into
Conversation
Tops the dedicated sweeper address up with gas when it falls below the configured low-water mark: derive the address, read its balance through the EVM RPC canister, burn ckETH from the minter's fee subaccount, and only then queue the transfer as a SweeperFundingRequest. Everything after the burn is the existing withdrawal pipeline. Conservative at every failure point: an unreadable balance is never treated as "empty" — that would burn ckETH for gas already in place — and a failed burn halts funding, and eventually sweeping, rather than spending un-burned ETH. Honouring "offset against subsequent burns" forced a data-model change: `SweeperFundingRequest` now carries `cketh_burned` distinct from `withdrawal_amount`. With a single field the offset is self-defeating — burning the reduced amount also reduces the transfer, so the credit grows forever instead of being consumed. `burn_for` is floored at the ledger minimum so every funding has a real burn (its index keys the whole pipeline) and so a credit larger than the amount due cannot strand funding; flooring only ever burns more than strictly required, which is the safe direction. The end-to-end test drives the production path against a real EVM with nothing mocked — real ckETH ledger, real EVM RPC canister, local anvil, tECDSA signature — and asserts the burn came out of the fee account and that the ETH debited never exceeds the ckETH burned for it. It earned its place immediately by finding two defects that unit tests cannot see, both timer-ordering rather than logic: - The install-time funding check raced the install-time key fetch, so it always skipped and the next attempt was a whole interval later — the very delay the immediate check exists to avoid. - Fixing that by fetching the key from the task was worse: a second concurrent `ecdsa_public_key` call trips an ic-cdk invariant and *traps the minter*. The task therefore reads the cached key and its first run is scheduled after a short delay instead. Two knock-on changes to shared test infrastructure: - The mocked harness could not parse `eth_getBalance` at all, so a new RPC method broke every mocked test. It now knows the method and answers the install-time funding check once with a topped-up balance, since an unanswered outcall stays in flight forever there, skews assertions on pending outcalls and prevents the canister from stopping. - `should_be_able_to_stop_canister_during_scraping` counted *all* pending outcalls to prove scraping was in flight; it now counts only `eth_getLogs`, so it is no longer coupled to whatever other periodic task happens to be due. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds periodic ckETH sweeper fee funding through the existing withdrawal pipeline.
Changes:
- Adds balance-based funding, burn-first accounting, and in-flight protection.
- Separates transferred ETH from ckETH burned in events and requests.
- Adds mock support, unit tests, and a live end-to-end Anvil test.
Reviewed changes
Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
rs/ethereum/cketh/test_utils/src/sweeper_funding.rs |
Adds the live funding test harness. |
rs/ethereum/cketh/test_utils/src/mock.rs |
Supports and filters balance RPC calls. |
rs/ethereum/cketh/test_utils/src/lib.rs |
Settles install-time funding checks. |
rs/ethereum/cketh/test_utils/src/anvil.rs |
Adds mainnet-like Anvil and balance helpers. |
rs/ethereum/cketh/minter/tests/sweeper_funding.rs |
Tests funding end to end. |
rs/ethereum/cketh/minter/tests/dump_stable_memory.rs |
Maps the new burn field. |
rs/ethereum/cketh/minter/tests/cketh.rs |
Filters scraping outcall assertions. |
rs/ethereum/cketh/minter/src/sweeper/tests.rs |
Tests funding plans and concurrency guards. |
rs/ethereum/cketh/minter/src/sweeper.rs |
Implements the funding task. |
rs/ethereum/cketh/minter/src/state/transactions/tests.rs |
Updates funding fixtures. |
rs/ethereum/cketh/minter/src/state/transactions/mod.rs |
Separates burn and transfer amounts. |
rs/ethereum/cketh/minter/src/state/tests.rs |
Updates state generators and fixtures. |
rs/ethereum/cketh/minter/src/state/sweeper_funding/tests.rs |
Tests in-flight accounting. |
rs/ethereum/cketh/minter/src/state/sweeper_funding.rs |
Tracks earmarked funding. |
rs/ethereum/cketh/minter/src/state/audit/tests.rs |
Updates event mapping tests. |
rs/ethereum/cketh/minter/src/state/audit.rs |
Replays burn and in-flight state. |
rs/ethereum/cketh/minter/src/state.rs |
Finalizes funding accounting. |
rs/ethereum/cketh/minter/src/main.rs |
Registers funding timers and exposes events. |
rs/ethereum/cketh/minter/src/lib.rs |
Exports sweeper logic and intervals. |
rs/ethereum/cketh/minter/src/endpoints.rs |
Exposes the burn amount in events. |
rs/ethereum/cketh/minter/cketh_minter.did |
Updates the Candid event schema. |
rs/ethereum/cketh/minter/BUILD.bazel |
Adds the long-running test target. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| ic_cdk_timers::set_timer(INITIAL_SWEEPER_FUNDING_DELAY, async { | ||
| fund_sweeper_address().await; | ||
| }); |
There was a problem hiding this comment.
Fixed in 25a99e0. You are right, and the distinction you are drawing is the one my commit message glossed over: a delay narrows the window, it does not order the two.
The first check now runs from the same timer callback as the key fetch, immediately after it:
ic_cdk_timers::set_timer(Duration::from_secs(0), async {
let _ = lazy_call_ecdsa_public_key().await;
fund_sweeper_address().await;
});The ordering is structural rather than timed, and INITIAL_SWEEPER_FUNDING_DELAY is gone. This also keeps the property the delay was introduced for in the first place: the two stay on one task, so there is never a second concurrent ecdsa_public_key call — which trips an ic-cdk invariant and traps the minter. That was the defect the delay was papering over, and sequencing addresses both at once.
On your second scenario — a key fetch that traps, leaving periodic funding to skip until something else populates the key — that remains true, and I decided against guarding it. lazy_call_ecdsa_public_key traps rather than returning an error, so it cannot be retried in place, and re-arming a short retry from the funding task would be protecting a state where the feature cannot work anyway: without the master public key there is no sweeper address to fund, so sweeping is blocked by the missing key rather than by the funding schedule. Recorded in the commit message rather than left implicit.
One knock-on worth mentioning: the mocked fixture no longer advances IC time by 60 seconds to reach the check. That jump was itself a nuisance — it perturbs the other periodic timers, which is why the fixture deliberately does not re-settle after upgrade_minter — so removing it takes out a source of test coupling. I ran the three mocked suites locally to confirm the fixture still settles the check without it (cketh_test, ckerc20_test, fee_account_test, all passing), since this changes install-time timer ordering for every mocked test.
The initial check ran on a 60-second timer, which does not order it after the `ecdsa_public_key` call it depends on — it only guesses that 60 seconds is enough. If the key took longer, the check found no cached key, returned, and the next attempt was a whole 24-hour interval away: exactly the delay an immediate check exists to avoid. Run it from the same timer callback as the key fetch instead, immediately after it. The ordering is then structural rather than timed, and the arbitrary constant goes away. Keeping the two on one task also preserves the property the delay was originally introduced for: two concurrent `ecdsa_public_key` calls trap the canister. A key fetch that traps still leaves funding to the periodic tick. That is tolerable because the sweeper address cannot be derived without the key at all, so sweeping is blocked by the missing key rather than by the funding schedule. The mocked fixture no longer advances IC time, which also removes the 60-second jump that perturbed other periodic timers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 22 out of 22 changed files in this pull request and generated no new comments.
Suppressed comments (1)
rs/ethereum/cketh/minter/tests/sweeper_funding.rs:17
- This seeds the fee account only after
new_livehas installed the minter, but installation immediately starts the funding callback (test_utils/src/sweeper_funding.rs:92-95). The callback can therefore attempt the burn before this transfer, or burn between this transfer and thesupply_before/fee_account_beforesnapshots, making this long test nondeterministically fail or observe zero burned. Seed the fee account before installing the minter (for example, pass the initial balance into the setup constructor) so the task cannot race the test arrangement.
setup.mint_cketh(setup.fee_account(), FEE_ACCOUNT_BALANCE);
The live test funded the fee account after `new_live` had installed the minter, but installation starts the funding task. The task could therefore attempt its burn before that transfer landed — skipping, with the next attempt a whole interval away — or burn between the transfer and the test's snapshots, which reads as nothing having been burned. Nondeterministic either way. Sequencing the check after the key fetch made this more likely rather than less, since the check no longer waits a fixed delay before looking. Seed the fee account in the ledger's initial balances instead, so the balance exists before the minter is installed at all and the task cannot precede it. `new_live_with_empty_fee_account` covers the case that wants it empty. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mbjorkqvist
left a comment
There was a problem hiding this comment.
On the suppressed comment (tests/sweeper_funding.rs:17, seeding the fee account after install)
Confirmed and fixed in 3a2e797. Both failure modes you describe are real, and there is a connection worth recording: the ordering fix in the previous round made this race more likely, not less. The old 60-second delay had been accidentally giving the test time to seed the fee account before the task looked; sequencing the check straight after the key fetch removed that grace period.
The fee account is now seeded through the ledger's initial_balances, so the balance exists before the minter is installed at all and the task cannot precede it — your suggestion, and the right shape, since the ledger is installed first. new_live_with_empty_fee_account covers the one test that wants it empty.
Verified by running both live suites rather than reasoning about it, and that turned up a second instance of the same class which the fix had exposed:
should_not_fund_a_sweeper_above_the_low_water_mark (in PR 7 of this stack) failed — supply 1.0 → 0.9 ETH, a burn where the test asserts none. With the fee account funded from install and the check firing immediately, the install-time check funded the sweeper before the test could top it up. And there is no second chance inside the test window: the next scheduled check is 24 hours away.
That test now arranges itself in the only order that is deterministic:
- start with an empty fee account, so the install-time check decides a funding is due, fails to burn, and changes nothing — the address is undiscoverable until the minter caches its key, so this is the only window;
- read the sweeper address, set its balance above the low-water mark, then fund the fee account;
- upgrade the minter to re-arm the timers, so a check runs again inside the test;
- assert no burn — which now means it declined with a funded fee account and a real opportunity to act.
Both suites pass: sweeper_funding 370 s, sweeper_funding_hardening 492 s. The other two hardening tests needed no change — one deliberately uses the empty fee account, and the revert test's set_code has the full 6-minute withdrawal-timer window before the transfer is sent, so its burn landing early is expected rather than racy.
Part of DEFI-2933 (sweeper fee funding), fifth of a seven-PR stack. Targets #11083.
What
The periodic task that keeps the dedicated sweeper address funded: derive the address, read its balance through the EVM RPC canister, burn ckETH from the minter's fee subaccount, and only then queue the transfer. Everything after the burn is the existing withdrawal pipeline.
Conservative at every failure point. An unreadable balance is never treated as "empty" — that would burn ckETH for gas already in place — and a failed burn halts funding, and eventually sweeping, rather than moving ETH nothing has paid for.
A data-model change the accounting forced
Honouring "offset against subsequent burns" required
SweeperFundingRequestto carry the burn separately from the amount it moves. With a single field the offset is self-defeating: reducing the burn reduces the transfer by the same amount, so the credit is never consumed and grows forever. The request now records both, and the accounting records only the burn.The burn is floored at the ledger minimum so every funding has a real burn index — the whole pipeline is keyed on it — and so a credit larger than the amount due draws down instead of stranding funding. Flooring only ever burns more than strictly required, which is the safe direction.
The end-to-end test
Drives the production path against a real EVM with nothing mocked: real ckETH ledger, real EVM RPC canister, local anvil, threshold-ECDSA signature. It asserts the burn came out of the fee account and that the ETH debited never exceeds the ckETH burned for it.
It earned its place immediately by finding two defects that unit tests cannot see, both timer-ordering rather than logic:
ecdsa_public_keycall trips an ic-cdk invariant and traps the minter. The task now reads the cached key, and its first run is scheduled after a short delay instead.Knock-on changes to shared test infrastructure
eth_getBalanceat all, so a new RPC method broke every mocked test. It now knows the method and answers the install-time check once with a topped-up balance, since an unanswered outcall stays in flight forever, skews assertions on pending outcalls and stops the minter from stopping. It deliberately does not re-settle after an upgrade: advancing time to that check perturbs other periodic timers and breaks unrelated tests, andstop_minterdrains pending outcalls anyway.should_be_able_to_stop_canister_during_scrapingcounted all pending outcalls to prove scraping was in flight; it now counts onlyeth_getLogs, so it is no longer coupled to whatever other periodic task happens to be due.Stack
Merge in order; each PR targets the one above it.