Surface failed and pending payments across backends

[benthecarman]

Summary

Keep failed and pending sends visible in the transaction history across Lightning, Spark, and Cashu, and resolve sends whose outcome was unknown instead of leaving them pending forever.

Stacked on #107 and targets its branch until it merges. The first commit is #89's; this PR supersedes #89 and carries it forward on top of #107.

Changes

Surface failed and pending outbound Lightning payments in list_transactions. ldk-node records a synchronously failed send as a Failed outbound record before returning the error, and it was hidden by the completed-only filter. Outbound attempts now always surface; unpaid inbound invoices stay hidden. A failed internal MPP leg no longer lists as a separate payment.

Persist submitted trusted payments. Spark and Cashu sends stay visible while pending and after a confirmed failure, even when the backend has no record. Backend terminal records take precedence and the local copy is pruned once the backend lists the payment. History listings and rebalance checks pass the backend list through untouched when nothing is recorded locally.

Reconcile uncertain Cashu melts. Interrupted melts are finalized through the CDK saga log at startup and whenever a melt ends without a definite result; submitted payments the CDK has no saga for are checked against the mint. Every outcome, including a reconciled one, reports to the rebalance watcher, so a rebalance waits for the real result. An unpaid quote is melted again after a transient error; the CDK and the mint reject a quote that is actually in flight.

Harden the MPP path. A duplicate invoice is rejected before the trusted leg goes out, since that leg cannot be recalled. The Lightning leg is found among recent outbound records by hash and direction, reading only the newest page of the payment list, rather than by deriving an LDK payment ID from the hash.

Tests

Unit tests for the payment store (restart survival, backend precedence, pruning, pass-through, corrupt records) and the error classifiers. Integration tests: a failed Lightning send lists once as failed; a failed MPP leg does not list separately; a repeated MPP invoice keeps the completed payment; a failed Cashu send is surfaced. Both full integration suites pass on the rebased branch.

🤖 Generated with Claude Code

[hash-money]

Thanks @benthecarman — agreed, this is the right shape, and closing #89 in favour of it is the right call. Surfacing failed and pending sends consistently across all three backends, reconciling uncertain melts through the saga log, and refusing the MPP trusted leg for a duplicate invoice closes the asymmetry we'd been uneasy about; carrying #89's Lightning commit forward on top is fine by us.

We pulled 266a770 under our product layer (wallet-sdk + wallet-ffi on top of orange-sdk: VSS-backed store, Cashu on Mutinynet signet) and ran it — the PR targets #107's branch, so there's no CI run on it yet:

• Build / tests: 422 of our unit + integration tests pass against it, clippy -D warnings clean. (The one failure was our own redactor's wall-clock bound tripping on a loaded box; it passes alone — unrelated.) The workspace needed a lock refresh to take esplora-client 0.13 → bitreq 0.3.7 via ldk-node 13ac606.
• On-device (Galaxy A12, Android 13; live wallet with three channels and a Cashu float; VSS-backed store; live cdk-mintd): installed over the existing wallet; balances and history intact across the ldk-node 0cea341 → 13ac606 move; steady state after startup is quiet.

Two notes from the diff. The first we think is worth a small change; the second is data.

1. Failed trusted sends are never pruned from PaymentStore, and every record is re-read at startup. merge() drops a local record only when the backend lists that payment with a non-Pending status (payment_store.rs:160-166). For Cashu that never happens for a failure: the CDK records a melt as a Transaction only on Paid — cdk-0.16.0/src/wallet/melt/mod.rs:797-805 (add_transaction_for_pending_melt, gated on new_state == Paid) and src/wallet/melt/saga/mod.rs:163-168 (after State::Spent; the Unpaid/Failed branch goes to [compensated], which releases the quote and proofs and writes no transaction) — and Cashu::list_payments is a straight map of list_transactions (cashu/mod.rs:174-188). So each failed melt leaves a Failed record that persists indefinitely, and PaymentStore::new lists and reads all of them on every start (payment_store.rs:59-61). On an in-process store that's noise; on a VSS-backed store (ours — the cold-start path from #90) it's one remote read per historical failed send, forever. Suggestion: drop Failed records at load or in merge once they're older than some horizon (past the newest backend record, or a fixed age), so the store stays bounded to unresolved work. Happy to send that as a small follow-up on top of this branch if you'd rather not fold it in.

2. Data, no ask — two things we measured rather than reasoned about:

• A trusted send now does two sequential KV writes before pay() returns: insert_pending → persist (cashu/mod.rs:704, spark/mod.rs:227 → payment_store.rs:134-135) and then tx_metadata.upsert (lib.rs:1350). Durable-before-background is the right order; on VSS each is a round trip, so we'll keep an eye on it. Nothing to change.

• Cold start on the phone, interleaved A/B/A (first VSS request → balance ready), same wallet, minutes apart:

  build	init → LDK node build	LDK node build → node up	total
  our pin ed29c6c (n=2)	4.9 / 5.3 s	15.4 / 8.3 s	21.0 / 14.5 s
  266a770 (n=3)	6.4 / 7.0 / 6.6 s	9.7 / 17.8 / 18.2 s	17.0 / 25.3 / 25.7 s

  The middle column is our Cold start: single VssStore serves all Cashu + LDK reads synchronously — 19s vs 0.56s local (34x) #90 problem (≈100 sequential VSS reads — unchanged here, and the noise source). The consistent +1.5 s in the first column is the one new thing before the node comes up: finalize_pending_melts is the only startup marker not already present on ed29c6c (check_all_mint_quotes, get_active_mint_quotes, the auth-keyset refresh all are), and it ran before Creating LDK node in all three starts. On this wallet that pass includes a status check for a stale saga from an old interrupted melt whose mint response cdk can't parse (missing field 'quote'), so it's rechecked at every backoff tick — a CDK/mint matter, not this PR. VSS ops at startup went 94 GET / 10 LIST → 103 / 13.

For the record, we checked the page assumption behind recent_outbound_bolt11 (lightning_wallet.rs:309-312): ldk-node 13ac606 documents the page as newest-first (src/payment/store.rs:49), so the just-inserted failed MPP leg is always on it, and a duplicate older than that page falls through to ldk-node's own DuplicatePayment as your comment says. Fine by us.

We'll bump our pin when #107 lands and take this on top when it merges. Our staging capture is scripted — point us at a branch if another run is useful.

[benthecarman]

@hash-money can you do a test with these changes + all the bitreq fixes? This should slightly effect start up time but make cashu payments faster

[hash-money]

Will do. Scope I'll run, so you can correct it: 266a770 (#108 on #107) plus the bitreq fixes as I understand them — the 0.3.7 that ldk-node 13ac606 pulls in via esplora-client 0.13, with our flush-after-write fix (corepc#705) applied on top, since that one is still open. Device: the same Galaxy A12 / VSS-backed staging wallet as above. Measurements: cold start A/B/A against our current pin (same method as the table above), and a Cashu payment timed from the app's log — mint quote → melt/receive settled. I'll post the before/after here within the day.

[hash-money]

Done — 266a770 (#108 on #107) + bitreq 0.3.7 with the corepc#705 flush fix applied on top (verified in the lock: bitreq v0.3.7 (path: bitreq-0.3.7-flush) via esplora-client 0.13 ← ldk-node 13ac606). Baseline is our pin ed29c6c on the same builds of everything else.

Cashu payments (desktop, same wallet-sdk/orange-sdk code as the app, local SQLite store, live cdk-mintd on Mutinynet). Counterparty = a second wallet on our LSP with a JIT channel; receives are its LN payment into the Cashu wallet's mint quote, sends are the Cashu wallet melting to pay its LN invoice. Wall-clock from the HTTP call to the terminal event:

build	receive 2,000 sats (quote → credited)	send 500 sats (call → PaymentSuccessful)
ed29c6c	1.8 s · 3.0 s (first 20,000-sat funding: ~2.5 s)	2.2 s · 1.8 s · 1.8 s
266a770 + bitreq flush	1.7 s · 2.7 s · 2.8 s	2.6 s · 2.0 s · 1.9 s

No measurable difference on a local store — both are dominated by the Lightning leg and the mint round-trips (post_melt_quote → post_melt → PaymentSuccessful is ~1–2 s in both logs). Daemon startup on the local store with the new build: ~1.0 s to /test/status ok.

Cold start on the phone (Galaxy A12, VSS-backed store, same wallet, interleaved). First VSS request → balance ready:

build	run 1	run 2	run 3	VSS ops to ready
ed29c6c	20.5 s	12.3 s	—	94 GET / 10 LIST
266a770 + bitreq flush	25.0 s	26.8 s	19.8 s	103 GET / 13 LIST

Same shape as the table above from yesterday (14.5/21.0 vs 17.0/25.3/25.7): roughly +5 s on average with ±7 s run-to-run spread, all of it in the ~100 sequential VSS reads during the LDK node build (#90), which go over hyper/reqwest — so the bitreq flush fix cannot reach it. The consistent new item is the startup finalize_pending_melts pass (+1.5 s before the node is created).

Cashu receive on the phone (counterparty pays the phone's mint quote; timed from the phone log):

The phone has Lightning inbound, so orange-sdk routed the 1,000-sat receives to Lightning, not Cashu (Trusted stayed at 2,404; that's the tier router working as designed) — so I could not get a Cashu receive on the phone tonight; the Cashu numbers above are the desktop ones. What the phone did give is the Lightning receive latency #107 targets, HTLC commitment_signed → PaymentReceived on the phone, payer = the same counterparty over our LSP:

build	phone-side HTLC → event	payer's pay-invoice call
ed29c6c	4.3 s (n=1; the other sample's log window missed the markers)	5.6 s · 5.6 s
266a770 + bitreq flush	6.0 s · 5.6 s	7.4 s · 7.0 s

Direction is slower by ~1.5–2 s on this device with n=2, which is inside what a VSS-persist-gated receive (each commitment step is a synchronous remote persist here) can swing; I'd treat it as "no improvement seen", not a regression claim.

So: startup on VSS is ~5 s slower on average in our sample (the VSS read count went 94→103, and the startup melt reconciliation is on the critical path); Cashu payment latency unchanged within noise on our setup. Happy to rerun with a different bitreq state if I've misread "all the bitreq fixes".