Skip to content

feat(cketh): sweeper fee-funding task, with an end-to-end test - #11086

Draft
mbjorkqvist wants to merge 5 commits into
mathias/DEFI-2933-burn-first-accountingfrom
mathias/DEFI-2933-funding-task
Draft

feat(cketh): sweeper fee-funding task, with an end-to-end test#11086
mbjorkqvist wants to merge 5 commits into
mathias/DEFI-2933-burn-first-accountingfrom
mathias/DEFI-2933-funding-task

Conversation

@mbjorkqvist

@mbjorkqvist mbjorkqvist commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Part of DEFI-2933 (sweeper fee funding), fifth of a seven-PR stack. Targets #11083.

What

The periodic task that keeps the dedicated sweeper address funded: derive the address, read its balance through the EVM RPC canister, burn ckETH from the minter's fee subaccount, and only then queue the transfer. Everything after the burn is the existing withdrawal pipeline.

Conservative at every failure point. An unreadable balance is never treated as "empty" — that would burn ckETH for gas already in place — and a failed burn halts funding, and eventually sweeping, rather than moving ETH nothing has paid for.

A data-model change the accounting forced

Honouring "offset against subsequent burns" required SweeperFundingRequest to carry the burn separately from the amount it moves. With a single field the offset is self-defeating: reducing the burn reduces the transfer by the same amount, so the credit is never consumed and grows forever. The request now records both, and the accounting records only the burn.

The burn is floored at the ledger minimum so every funding has a real burn index — the whole pipeline is keyed on it — and so a credit larger than the amount due draws down instead of stranding funding. Flooring only ever burns more than strictly required, which is the safe direction.

The end-to-end test

Drives the production path against a real EVM with nothing mocked: real ckETH ledger, real EVM RPC canister, local anvil, threshold-ECDSA signature. It asserts the burn came out of the fee account and that the ETH debited never exceeds the ckETH burned for it.

It earned its place immediately by finding two defects that unit tests cannot see, both timer-ordering rather than logic:

  • The install-time funding check raced the install-time key fetch, so it always skipped and the next attempt was a whole interval away — exactly the delay an immediate check exists to avoid.
  • Fixing that by fetching the key from the task was worse: a second concurrent ecdsa_public_key call trips an ic-cdk invariant and traps the minter. The task now reads the cached key, and its first run is scheduled after a short delay instead.

Note on CI: the new sweeper_funding target is tagged long_test (~6.5 minutes, nearly all of it waiting on the minter's 6-minute withdrawal timer), so ordinary CI skips it. It needs a run with CI_ALL_BAZEL_TARGETS before merge.

Knock-on changes to shared test infrastructure

  • The mocked harness could not parse eth_getBalance at all, so a new RPC method broke every mocked test. It now knows the method and answers the install-time check once with a topped-up balance, since an unanswered outcall stays in flight forever, skews assertions on pending outcalls and stops the minter from stopping. It deliberately does not re-settle after an upgrade: advancing time to that check perturbs other periodic timers and breaks unrelated tests, and stop_minter drains pending outcalls anyway.
  • should_be_able_to_stop_canister_during_scraping counted all pending outcalls to prove scraping was in flight; it now counts only eth_getLogs, so it is no longer coupled to whatever other periodic task happens to be due.

Stack

Merge in order; each PR targets the one above it.

# PR Status
1 #11060 — Read a native ETH balance via the EVM RPC canister ready for review
2 #11065 — Burn ckETH from the minter's own fee subaccount ready for review
3 #11072 — Add the SweeperFunding withdrawal-request variant ready for review
4 #11083 — Burn-first accounting for sweeper fee funding Copilot clean, CI green
5 Sweeper fee-funding task, with an end-to-end test this PR
6 #11094 — Sweeper funding observability and the prepaid-gas gate open
7 Adversarial end-to-end coverage of sweeper fee funding not yet opened

Tops the dedicated sweeper address up with gas when it falls below the
configured low-water mark: derive the address, read its balance through the EVM
RPC canister, burn ckETH from the minter's fee subaccount, and only then queue
the transfer as a SweeperFundingRequest. Everything after the burn is the
existing withdrawal pipeline.

Conservative at every failure point: an unreadable balance is never treated as
"empty" — that would burn ckETH for gas already in place — and a failed burn
halts funding, and eventually sweeping, rather than spending un-burned ETH.

Honouring "offset against subsequent burns" forced a data-model change:
`SweeperFundingRequest` now carries `cketh_burned` distinct from
`withdrawal_amount`. With a single field the offset is self-defeating — burning
the reduced amount also reduces the transfer, so the credit grows forever
instead of being consumed. `burn_for` is floored at the ledger minimum so every
funding has a real burn (its index keys the whole pipeline) and so a credit
larger than the amount due cannot strand funding; flooring only ever burns more
than strictly required, which is the safe direction.

The end-to-end test drives the production path against a real EVM with nothing
mocked — real ckETH ledger, real EVM RPC canister, local anvil, tECDSA
signature — and asserts the burn came out of the fee account and that the ETH
debited never exceeds the ckETH burned for it.

It earned its place immediately by finding two defects that unit tests cannot
see, both timer-ordering rather than logic:

- The install-time funding check raced the install-time key fetch, so it always
  skipped and the next attempt was a whole interval later — the very delay the
  immediate check exists to avoid.
- Fixing that by fetching the key from the task was worse: a second concurrent
  `ecdsa_public_key` call trips an ic-cdk invariant and *traps the minter*. The
  task therefore reads the cached key and its first run is scheduled after a
  short delay instead.

Two knock-on changes to shared test infrastructure:

- The mocked harness could not parse `eth_getBalance` at all, so a new RPC
  method broke every mocked test. It now knows the method and answers the
  install-time funding check once with a topped-up balance, since an unanswered
  outcall stays in flight forever there, skews assertions on pending outcalls
  and prevents the canister from stopping.
- `should_be_able_to_stop_canister_during_scraping` counted *all* pending
  outcalls to prove scraping was in flight; it now counts only `eth_getLogs`,
  so it is no longer coupled to whatever other periodic task happens to be due.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds periodic ckETH sweeper fee funding through the existing withdrawal pipeline.

Changes:

  • Adds balance-based funding, burn-first accounting, and in-flight protection.
  • Separates transferred ETH from ckETH burned in events and requests.
  • Adds mock support, unit tests, and a live end-to-end Anvil test.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
rs/ethereum/cketh/test_utils/src/sweeper_funding.rs Adds the live funding test harness.
rs/ethereum/cketh/test_utils/src/mock.rs Supports and filters balance RPC calls.
rs/ethereum/cketh/test_utils/src/lib.rs Settles install-time funding checks.
rs/ethereum/cketh/test_utils/src/anvil.rs Adds mainnet-like Anvil and balance helpers.
rs/ethereum/cketh/minter/tests/sweeper_funding.rs Tests funding end to end.
rs/ethereum/cketh/minter/tests/dump_stable_memory.rs Maps the new burn field.
rs/ethereum/cketh/minter/tests/cketh.rs Filters scraping outcall assertions.
rs/ethereum/cketh/minter/src/sweeper/tests.rs Tests funding plans and concurrency guards.
rs/ethereum/cketh/minter/src/sweeper.rs Implements the funding task.
rs/ethereum/cketh/minter/src/state/transactions/tests.rs Updates funding fixtures.
rs/ethereum/cketh/minter/src/state/transactions/mod.rs Separates burn and transfer amounts.
rs/ethereum/cketh/minter/src/state/tests.rs Updates state generators and fixtures.
rs/ethereum/cketh/minter/src/state/sweeper_funding/tests.rs Tests in-flight accounting.
rs/ethereum/cketh/minter/src/state/sweeper_funding.rs Tracks earmarked funding.
rs/ethereum/cketh/minter/src/state/audit/tests.rs Updates event mapping tests.
rs/ethereum/cketh/minter/src/state/audit.rs Replays burn and in-flight state.
rs/ethereum/cketh/minter/src/state.rs Finalizes funding accounting.
rs/ethereum/cketh/minter/src/main.rs Registers funding timers and exposes events.
rs/ethereum/cketh/minter/src/lib.rs Exports sweeper logic and intervals.
rs/ethereum/cketh/minter/src/endpoints.rs Exposes the burn amount in events.
rs/ethereum/cketh/minter/cketh_minter.did Updates the Candid event schema.
rs/ethereum/cketh/minter/BUILD.bazel Adds the long-running test target.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread rs/ethereum/cketh/minter/src/main.rs Outdated
Comment on lines +117 to +119
ic_cdk_timers::set_timer(INITIAL_SWEEPER_FUNDING_DELAY, async {
fund_sweeper_address().await;
});

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 25a99e0. You are right, and the distinction you are drawing is the one my commit message glossed over: a delay narrows the window, it does not order the two.

The first check now runs from the same timer callback as the key fetch, immediately after it:

ic_cdk_timers::set_timer(Duration::from_secs(0), async {
    let _ = lazy_call_ecdsa_public_key().await;
    fund_sweeper_address().await;
});

The ordering is structural rather than timed, and INITIAL_SWEEPER_FUNDING_DELAY is gone. This also keeps the property the delay was introduced for in the first place: the two stay on one task, so there is never a second concurrent ecdsa_public_key call — which trips an ic-cdk invariant and traps the minter. That was the defect the delay was papering over, and sequencing addresses both at once.

On your second scenario — a key fetch that traps, leaving periodic funding to skip until something else populates the key — that remains true, and I decided against guarding it. lazy_call_ecdsa_public_key traps rather than returning an error, so it cannot be retried in place, and re-arming a short retry from the funding task would be protecting a state where the feature cannot work anyway: without the master public key there is no sweeper address to fund, so sweeping is blocked by the missing key rather than by the funding schedule. Recorded in the commit message rather than left implicit.

One knock-on worth mentioning: the mocked fixture no longer advances IC time by 60 seconds to reach the check. That jump was itself a nuisance — it perturbs the other periodic timers, which is why the fixture deliberately does not re-settle after upgrade_minter — so removing it takes out a source of test coupling. I ran the three mocked suites locally to confirm the fixture still settles the check without it (cketh_test, ckerc20_test, fee_account_test, all passing), since this changes install-time timer ordering for every mocked test.

The initial check ran on a 60-second timer, which does not order it after the
`ecdsa_public_key` call it depends on — it only guesses that 60 seconds is
enough. If the key took longer, the check found no cached key, returned, and the
next attempt was a whole 24-hour interval away: exactly the delay an immediate
check exists to avoid.

Run it from the same timer callback as the key fetch instead, immediately after
it. The ordering is then structural rather than timed, and the arbitrary constant
goes away. Keeping the two on one task also preserves the property the delay was
originally introduced for: two concurrent `ecdsa_public_key` calls trap the
canister.

A key fetch that traps still leaves funding to the periodic tick. That is
tolerable because the sweeper address cannot be derived without the key at all,
so sweeping is blocked by the missing key rather than by the funding schedule.

The mocked fixture no longer advances IC time, which also removes the 60-second
jump that perturbed other periodic timers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated no new comments.

Suppressed comments (1)

rs/ethereum/cketh/minter/tests/sweeper_funding.rs:17

  • This seeds the fee account only after new_live has installed the minter, but installation immediately starts the funding callback (test_utils/src/sweeper_funding.rs:92-95). The callback can therefore attempt the burn before this transfer, or burn between this transfer and the supply_before/fee_account_before snapshots, making this long test nondeterministically fail or observe zero burned. Seed the fee account before installing the minter (for example, pass the initial balance into the setup constructor) so the task cannot race the test arrangement.
    setup.mint_cketh(setup.fee_account(), FEE_ACCOUNT_BALANCE);

The live test funded the fee account after `new_live` had installed the minter,
but installation starts the funding task. The task could therefore attempt its
burn before that transfer landed — skipping, with the next attempt a whole
interval away — or burn between the transfer and the test's snapshots, which
reads as nothing having been burned. Nondeterministic either way.

Sequencing the check after the key fetch made this more likely rather than less,
since the check no longer waits a fixed delay before looking.

Seed the fee account in the ledger's initial balances instead, so the balance
exists before the minter is installed at all and the task cannot precede it.
`new_live_with_empty_fee_account` covers the case that wants it empty.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@mbjorkqvist mbjorkqvist left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the suppressed comment (tests/sweeper_funding.rs:17, seeding the fee account after install)

Confirmed and fixed in 3a2e797. Both failure modes you describe are real, and there is a connection worth recording: the ordering fix in the previous round made this race more likely, not less. The old 60-second delay had been accidentally giving the test time to seed the fee account before the task looked; sequencing the check straight after the key fetch removed that grace period.

The fee account is now seeded through the ledger's initial_balances, so the balance exists before the minter is installed at all and the task cannot precede it — your suggestion, and the right shape, since the ledger is installed first. new_live_with_empty_fee_account covers the one test that wants it empty.

Verified by running both live suites rather than reasoning about it, and that turned up a second instance of the same class which the fix had exposed:

should_not_fund_a_sweeper_above_the_low_water_mark (in PR 7 of this stack) failed — supply 1.0 → 0.9 ETH, a burn where the test asserts none. With the fee account funded from install and the check firing immediately, the install-time check funded the sweeper before the test could top it up. And there is no second chance inside the test window: the next scheduled check is 24 hours away.

That test now arranges itself in the only order that is deterministic:

  1. start with an empty fee account, so the install-time check decides a funding is due, fails to burn, and changes nothing — the address is undiscoverable until the minter caches its key, so this is the only window;
  2. read the sweeper address, set its balance above the low-water mark, then fund the fee account;
  3. upgrade the minter to re-arm the timers, so a check runs again inside the test;
  4. assert no burn — which now means it declined with a funded fee account and a real opportunity to act.

Both suites pass: sweeper_funding 370 s, sweeper_funding_hardening 492 s. The other two hardening tests needed no change — one deliberately uses the empty fee account, and the revert test's set_code has the full 6-minute withdrawal-timer window before the transfer is sent, so its burn landing early is expected rather than racy.

@mbjorkqvist mbjorkqvist added the CI_ALL_BAZEL_TARGETS Runs all bazel targets label Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI_ALL_BAZEL_TARGETS Runs all bazel targets feat

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants