feat(minter): mint the pending deposits of credited sweeps on a timer

docs/design.md

@@ -363,9 +363,9 @@ The ckSOL minter then fetches the transaction with `getTransaction` and *settles
 
 Once settled, the balance of the main account is increased by the amount received. The transaction fee is borne by the batch: for each deposit in the transaction, the ckSOL minter enqueues a *pending mint* of the sweepable amount minus the deposit's share of the fee, `ceil(fee / k)`, to the user's account. Rounding the share up guarantees that the total of the pending mints is never larger than the amount received on the main account, so the ckSOL supply is always covered. The amount to mint of each deposit is recorded in the event together with the amount received, so that replaying the event log restores the pending mints as they were enqueued rather than recomputing them.
 
-Pending mints are processed on a timer: each one is a single `icrc1_transfer` call whose memo contains the sweep signature. The `created_at_time` of the transfer is fixed when the pending mint is enqueued and stored with it, so that every retry sends exactly the same arguments and the ckSOL ledger deduplicates it: a mint whose ledger call fails stays in the queue and is retried on the next run, and a `Duplicate` reply from the ledger is recorded as a successful mint with the block index it carries. Since the ledger only deduplicates transfers within its 24-hour window, a pending mint older than that is quarantined instead of retried. The sweep transaction itself is never resubmitted once it is finalized.
+Pending mints are processed on a timer: each one is a single `icrc1_transfer` call whose memo contains the sweep signature and the deposit id. The `created_at_time` of the transfer is fixed when the pending mint is enqueued and stored with it, so that every retry sends exactly the same arguments and the ckSOL ledger deduplicates it: a mint whose ledger call fails stays in the queue and is retried on the next run, and a `Duplicate` reply from the ledger is recorded as a successful mint with the block index it carries. Since the ledger only deduplicates transfers within its 24-hour window, a pending mint older than that is quarantined instead of retried. The sweep transaction itself is never resubmitted once it is finalized.
 
-A user can follow the progress with `deposit_status`, which takes a deposit id and reports its status: `Queued` with the sweepable amount, `Swept` and `Finalized` with the sweep signature, `Minted` with the ledger block index, and `Dropped` and `Quarantined` with the sweep signature when there is one. Since every sweep has its own id, a `Minted` status stays observable after the account has been swept again; the mint itself is also visible on the ckSOL ledger, whose memo links it to the sweep signature.
+A user can follow the progress with `deposit_status`, which takes a deposit id and reports its status: `Queued` with the sweepable amount, `Swept` and `Finalized` with the sweep signature, `Minted` with the ledger block index and the minted amount, and `Dropped` and `Quarantined` with the sweep signature when there is one. Since every sweep has its own id, a `Minted` status stays observable after the account has been swept again; the mint itself is also visible on the ckSOL ledger, whose memo links it to the sweep signature and the deposit id.
 
 Two failure cases exist before the transaction is finalized, and neither is retried by the ckSOL minter:
 

integration_tests/Cargo.toml

@@ -10,6 +10,8 @@ license.workspace = true
 [dependencies]
 assert_matches = { workspace = true }
 async-trait = { workspace = true }
+base64 = { workspace = true }
+bincode = { workspace = true }
 candid = { workspace = true }
 canlog = { workspace = true }
 cksol-types = { path = "../libs/types" }
@@ -31,10 +33,14 @@ sol_rpc_client = { workspace = true }
 sol_rpc_types = { workspace = true }
 solana-address = { workspace = true }
 solana-client = { workspace = true }
+solana-hash = { workspace = true }
 solana-keypair = { workspace = true }
+solana-message = { workspace = true }
 solana-native-token = { workspace = true }
 solana-signature = { workspace = true }
+solana-system-interface = { workspace = true, features = ["bincode"] }
 solana-system-transaction = { workspace = true }
+solana-transaction = { workspace = true, features = ["bincode"] }
 tokio = { workspace = true }
 
 [dev-dependencies]

integration_tests/src/fixtures.rs

@@ -1,5 +1,6 @@
-use crate::Setup;
+use crate::{Setup, validator::FEE_PER_SIGNATURE};
 use async_trait::async_trait;
+use base64::{Engine, engine::general_purpose::STANDARD};
 use cksol_types::{GetDepositAddressArgs, ProcessDepositArgs, Signature};
 use ic_pocket_canister_runtime::{
     ExecuteHttpOutcallMocks, JsonRpcRequestMatcher, JsonRpcResponse, MockHttpOutcalls,
@@ -222,6 +223,16 @@ impl MockBuilder {
             .check_signature_statuses(get_signature_statuses_finalized_response())
     }
 
+    /// Mock for `getTransaction` for a sweep of [`DEFAULT_CALLER_DEPOSIT_ADDRESS`] under the
+    /// given signature, reporting metadata that credits the minter's main account with the
+    /// sweepable amount minus the transaction fee of one signature.
+    pub fn get_sweep_transaction(self, signature: &Signature, sweepable_amount: Lamport) -> Self {
+        self.expect(
+            get_transaction_request(&signature.to_string()),
+            sweep_transaction_response(sweepable_amount),
+        )
+    }
+
     fn check_signature_statuses(self, response: JsonRpcResponse) -> Self {
         self.expect(get_signature_statuses_request(), response)
     }
@@ -310,6 +321,61 @@ pub fn get_deposit_transaction_response() -> JsonRpcResponse {
     }))
 }
 
+/// JSON-RPC `getTransaction` response for a sweep transaction moving the sweepable
+/// amount of [`DEFAULT_CALLER_DEPOSIT_ADDRESS`], minus the fee it pays as the only
+/// signer, to [`MINTER_ADDRESS`].
+fn sweep_transaction_response(sweepable_amount: Lamport) -> JsonRpcResponse {
+    const MAIN_BALANCE_BEFORE_SWEEP: Lamport = 5_000_000_000;
+    let deposit_address: Address = DEFAULT_CALLER_DEPOSIT_ADDRESS.parse().unwrap();
+    let transfer_amount = sweepable_amount - FEE_PER_SIGNATURE;
+    let message = solana_message::Message::new_with_blockhash(
+        &[solana_system_interface::instruction::transfer(
+            &deposit_address,
+            &MINTER_ADDRESS,
+            transfer_amount,
+        )],
+        Some(&deposit_address),
+        &solana_hash::Hash::default(),
+    );
+    let transaction = solana_transaction::versioned::VersionedTransaction::from(
+        solana_transaction::Transaction::new_unsigned(message),
+    );
+    let encoded_transaction = STANDARD.encode(
+        bincode::serialize(&transaction).expect("serializing the transaction should succeed"),
+    );
+    JsonRpcResponse::from(json!({
+        "jsonrpc": "2.0",
+        "result": {
+            "blockTime": 1700000000_i64,
+            "meta": {
+                "computeUnitsConsumed": 150,
+                "err": null,
+                "fee": FEE_PER_SIGNATURE,
+                "innerInstructions": [],
+                "loadedAddresses": { "readonly": [], "writable": [] },
+                "logMessages": [],
+                "postBalances": [
+                    RENT_EXEMPTION_THRESHOLD,
+                    MAIN_BALANCE_BEFORE_SWEEP + transfer_amount,
+                    1
+                ],
+                "postTokenBalances": [],
+                "preBalances": [
+                    RENT_EXEMPTION_THRESHOLD + sweepable_amount,
+                    MAIN_BALANCE_BEFORE_SWEEP,
+                    1
+                ],
+                "preTokenBalances": [],
+                "rewards": [],
+                "status": { "Ok": null }
+            },
+            "slot": 350_000_000_u64,
+            "transaction": [encoded_transaction, "base64"]
+        },
+        "id": 1
+    }))
+}
+
 fn get_balance_request() -> JsonRpcRequestMatcher {
     JsonRpcRequestMatcher::with_method("getBalance")
 }

integration_tests/src/ledger_init_args.rs

@@ -1,11 +1,16 @@
 //! Since the init args for the ICRC1 ledger are not published in a public crate,
 //! redefine them here to initialize the canister correctly.
 use candid::{CandidType, Deserialize, Nat, Principal};
-use cksol_types::MAX_SERIALIZED_MEMO_BYTES;
 use icrc_ledger_types::{icrc::generic_value::Value, icrc1::account::Account};
 use serde::Serialize;
 
 pub const LEDGER_TRANSFER_FEE: u64 = 50;
+/// The `max_memo_length` the ckSOL ledger is deployed with, matching the value most
+/// ICRC-1 ledgers use.
+///
+/// One byte below the largest memo the minter can produce, so the mint of a deposit
+/// id beyond `u32::MAX` would not fit. Raising it takes a ledger upgrade.
+const LEDGER_MAX_MEMO_BYTES: u16 = 80;
 const NNS_ROOT_PRINCIPAL: Principal = Principal::from_slice(&[0_u8, 0, 0, 0, 0, 0, 0, 3, 1, 1]);
 const FEE_COLLECTOR_SUBACCOUNT: [u8; 32] = [
     0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0x0f,
@@ -42,7 +47,7 @@ pub fn ledger_init_args(
             cycles_for_archive_creation: Some(100_000_000_000_000),
             max_transactions_per_response: None,
         },
-        max_memo_length: Some(MAX_SERIALIZED_MEMO_BYTES),
+        max_memo_length: Some(LEDGER_MAX_MEMO_BYTES),
         feature_flags: None,
         index_principal: None,
     })

integration_tests/src/lib.rs

@@ -91,6 +91,9 @@ impl SetupBuilder {
 
 pub struct Setup {
     env: Option<PocketIc>,
+    /// Whether the instance produces blocks on its own, which decides how
+    /// [`Setup::advance_time_and_settle`] lets a timer round run.
+    auto_progress: bool,
     minter_canister_id: CanisterId,
     ledger_canister_id: CanisterId,
     sol_rpc_canister_id: CanisterId,
@@ -204,7 +207,8 @@ impl Setup {
             None
         };
 
-        let env = if let PocketIcMode::LiveMode = make_live {
+        let auto_progress = matches!(make_live, PocketIcMode::LiveMode);
+        let env = if auto_progress {
             let mut env = env;
             let _ = env.make_live(None).await;
             env
@@ -218,6 +222,7 @@ impl Setup {
 
         Self {
             env: Some(env),
+            auto_progress,
             minter_canister_id,
             ledger_canister_id,
             sol_rpc_canister_id,
@@ -317,13 +322,38 @@ impl Setup {
         self.env.as_ref().unwrap().advance_time(duration).await
     }
 
-    /// Advances time and then lets the timers that became due complete their
-    /// HTTP outcalls. An outcall still in flight when time is advanced again
-    /// times out, and a timer needs several sequential outcalls per round.
+    /// Advances time and then lets the timers that became due complete their round.
+    ///
+    /// An instance that produces blocks on its own only needs wall-clock time for the
+    /// HTTP outcalls of the round to come back, and an outcall still in flight when
+    /// time is advanced again times out. An instance that does not produce blocks runs
+    /// nothing until it is ticked, and a round takes several sequential messages.
     pub async fn advance_time_and_settle(&self, duration: Duration) {
         const OUTCALL_SETTLE_DELAY: Duration = Duration::from_secs(2);
+        const TICKS_PER_ROUND: usize = 10;
+
         self.advance_time(duration).await;
-        tokio::time::sleep(OUTCALL_SETTLE_DELAY).await;
+        if self.auto_progress {
+            tokio::time::sleep(OUTCALL_SETTLE_DELAY).await;
+        } else {
+            for _ in 0..TICKS_PER_ROUND {
+                self.tick().await;
+            }
+        }
+    }
+
+    /// Advances time until the deposit is minted and returns the minted amount.
+    pub async fn wait_for_deposit_minted(&self, deposit_id: DepositSolId) -> Lamport {
+        const MAX_ITERATIONS: usize = 30;
+        for _ in 0..MAX_ITERATIONS {
+            if let DepositSolStatus::Minted { minted_amount, .. } =
+                self.minter().deposit_status(deposit_id).await
+            {
+                return minted_amount;
+            }
+            self.advance_time_and_settle(Duration::from_mins(1)).await;
+        }
+        panic!("Deposit {deposit_id} was not minted within timeout");
     }
 
     pub async fn execute_http_mocks(&self, mut mocks: impl ExecuteHttpOutcallMocks) {

integration_tests/tests/solana_test_validator.rs

@@ -345,5 +345,17 @@ async fn should_sweep_a_full_batch_of_deposits_in_one_transaction() {
         );
     }
 
+    for ((&deposit_id, &account), &deposit_amount) in
+        deposit_ids.iter().zip(&accounts).zip(&deposit_amounts)
+    {
+        let expected_minted_amount = deposit_amount - RENT_EXEMPTION_THRESHOLD - FEE_PER_SIGNATURE;
+        let minted_amount = setup.wait_for_deposit_minted(deposit_id).await;
+        assert_eq!(minted_amount, expected_minted_amount);
+        assert_eq!(
+            setup.ledger().balance_of(account).await,
+            expected_minted_amount
+        );
+    }
+
     setup.drop().await;
 }

integration_tests/tests/tests.rs

@@ -9,6 +9,7 @@ use cksol_int_tests::{
         default_process_deposit_args, deposit_transaction_signature,
         get_deposit_transaction_response,
     },
+    validator::FEE_PER_SIGNATURE,
 };
 use cksol_types::{
     DepositId, DepositSolArgs, DepositSolError, DepositSolStatus, DepositStatus,
@@ -1191,7 +1192,8 @@ mod deposit_sol_tests {
     }
 
     #[tokio::test]
-    async fn should_sweep_queued_deposit_after_timer() {
+    async fn should_sweep_finalize_and_mint_queued_deposit_after_timers() {
+        const SWEEPABLE_AMOUNT: Lamport = BALANCE_ABOVE_MINIMUM - RENT_EXEMPTION_THRESHOLD;
         let setup = SetupBuilder::new().with_proxy_canister().build().await;
         let deposit_id = setup
             .minter()
@@ -1228,6 +1230,31 @@ mod deposit_sol_tests {
             )));
         });
 
+        setup.advance_time(FINALIZE_TRANSACTIONS_DELAY).await;
+        setup
+            .execute_http_mocks(
+                MockBuilder::with_start_id(16)
+                    .finalize_transaction(SUBMISSION_BLOCK_HEIGHT)
+                    .get_sweep_transaction(&sweep_signature, SWEEPABLE_AMOUNT)
+                    .build(),
+            )
+            .await;
+
+        setup.wait_for_deposit_minted(deposit_id).await;
+
+        let minted_amount = SWEEPABLE_AMOUNT - FEE_PER_SIGNATURE;
+        assert_eq!(
+            setup.minter().deposit_status(deposit_id).await,
+            DepositSolStatus::Minted {
+                block_index: 0,
+                minted_amount,
+            }
+        );
+        assert_eq!(
+            setup.ledger().balance_of(DEFAULT_CALLER_ACCOUNT).await,
+            minted_amount
+        );
+
         setup.drop().await;
     }
 
@@ -1364,7 +1391,9 @@ mod deposit_sol_tests {
         let setup = setup
             .check_metrics()
             .await
-            .assert_contains_metric_matching(r"quarantined_swept_deposits 1 \d+")
+            .assert_contains_metric_matching(
+                r#"quarantined_swept_deposits\{cause="sweep_unreadable"\} 1 \d+"#,
+            )
             .into();
 
         let result = setup.minter().deposit_sol(DEFAULT_CALLER_ACCOUNT).await;

libs/types-internal/src/event.rs

@@ -150,6 +150,31 @@ pub enum EventType {
         /// The chain code used to derive subkeys (32 bytes).
         chain_code: Vec<u8>,
     },
+    /// The minter minted ckSOL on the ledger for a swept deposit whose sweep
+    /// was credited.
+    MintedSweptDeposit {
+        /// The identifier of the minted deposit.
+        deposit_id: u64,
+        /// The mint transaction index on the ckSOL ledger.
+        mint_block_index: u64,
+    },
+    /// The pending mint of a swept deposit cannot be retried: either it became
+    /// older than the 24-hour deduplication window of the ckSOL ledger, or the
+    /// ledger definitively rejected it. Retrying the transfer with the same
+    /// arguments fails forever, and fresh arguments could double mint.
+    ///
+    /// The deposit is quarantined to avoid any double minting and will not be
+    /// further processed without manual intervention.
+    ///
+    /// If the minter was down past the deduplication window, the underlying
+    /// transfer may nevertheless have landed on the ledger. Manual resolution
+    /// must therefore first search the ledger for a mint whose memo carries the
+    /// sweep signature before crediting by hand, otherwise a double mint
+    /// results.
+    QuarantinedPendingMint {
+        /// The identifier of the deposit whose pending mint was quarantined.
+        deposit_id: u64,
+    },
 }
 
 /// The mint enqueued for one deposit of a `CreditedSweep` event.