fix: vss client recovery and reset race - #1266
Conversation
|
Regtest APKDownload bitkit-dev-debug universal APK (expires in 30 days). |
2a079f7 to
68dbba4
Compare
68dbba4 to
a9e08f8
Compare
piotr-iohk
left a comment
There was a problem hiding this comment.
QA review from the diff and thread only — not run on device.
-
[gap]
VssBackupClientLdk.setup()still doesisSetup.completeExceptionally(it)without replacing the deferred — the same latch fixed here inVssBackupClient. Callers areclearNetworkGraphandresetPathfindingScores(LightningRepo.kt:665,:1904): one failed LDK-client setup makes every later "Reset network graph" / "Reset pathfinding scores" fail with the stale error and restart the node until the app is restarted. Same two-line fix; arguably in #1256's scope. -
[+1 greptile] The partial-wipe window (LDK deleted, keychain kept) is real but narrow — I checked the steps between
wipeStorageandkeychain.wipe()and onlyprivatePaykitAddressReservationRepo.clear()andkeychain.wipe()itself can throw. Still, if it happens the app presents a wallet whose channel state is gone. Consider callingresetWalletState()regardless of failure oncewipeStoragehas succeeded, or treating the remaining local wipes as non-fatal. -
[gap] Reset is now a long blocking operation with no UI feedback.
ResetAndRestoreScreennever clearsshowDialogon confirm, so the dialog stays up untilresetWalletState(), which now runs afterstop()— the full node build + initial sync, or minutes in the #1257 stop-hang case.rememberDebouncedClickonly debounces the tap; nothing stops a secondwipeWallet()mid-flight, andWipeWalletUseCasehas no re-entrancy guard, so the second run'sfinally { setWiping(false) }can clear the flag while the first is still wiping. Before this PR the onboarding switch was near-immediate so the window was tiny. Awipingflag on the dialog/button or a mutex in the use case would close it. -
[nit] If
stop()fails,wipeStorage().getOrThrow()aborts afterbackupRepo.reset()already stopped the observers and reset the VSS clients. Wallet survives, but backups are silently off until the nextRunningtransition (app restart). Pre-existing ordering, new abort point — worth restarting observers on that failure path.
@piotr-iohk Thanks, all four addressed in 2b5431e:
|
|
An agent is running and will check the Test 3 checkbox if it succeeds. |
piotr-iohk
left a comment
There was a problem hiding this comment.
Re-reviewed 2b5431e from the diff — not run on device. Items 1, 3 and the Greptile P1 look good; two new issues from the fix-up:
-
[blocker]
keychain.wipe()now runs beforeremovePublishedEndpointsForCleanup,removeBitkitPaymentEndpointsandcloseAndClear. All three go throughPaykitSdkService, whosePaykitSdkSessionProvider.loadSessionAccess()readsPAYKIT_SESSIONfrom the keychain on every call and returnsnullonce it is gone (liveSessionAccessis only returned when its secret matches the keychain value). SosyncPublicEndpoints(emptyList())/removePaykitReceiverMarker()run without a session, fail, and are swallowed — and thecontactSharingCleanupPendingflag is then erased bysettingsStore.reset(). Result: the wiped wallet's public endpoints and receiver marker stay published after Reset; on master they were removed. Every step betweenwipeStorageand the oldkeychain.wipe()position returnsResultor swallows (onlyprivatePaykitAddressReservationRepo.clear()can throw), so movingkeychain.wipe()back afterpubkyRepo.wipeLocalState()keeps Greptile's window closed in practice; wrapclear()if you want it airtight. -
[gap] The observer restart in
onFailureis dead code:backupRepo.startObservingBackups()runs while_isWipingis stilltrue(cleared infinally), so the guard added in this PR returns early with "Skipped observing backups while wiping". The unit test only passes becauseBackupRepois a mock. CallsetWiping(false)before the restart, or move the restart after thefinally.
@piotr-iohk Both addressed in 47187b3:
|
|
Handed off to |
47187b3 to
43449af
Compare
@piotr-iohk Both addressed in 43449af, which replaces the previous fix-up rather than moving lines around again. The wipe is now phased:
|
jvsena42
left a comment
There was a problem hiding this comment.
No blockers. Two LOW observations below, both pre-existing on v2.4.1 and neither introduced by this PR — I'm raising them only because they sit inside the window this PR's title claims to close, and in one case the PR body's wording is stronger than the code.
Neither is a reversal: greptile's guard, piotr's items 1-4 and the phased-wipe design in 43449af all stay as landed.
Checked and clean.
Exits from ResetAndRestoreScreen: system Back while wiping is swallowed by BackHandler(enabled = isWiping) {} (:76); top-bar back is hidden via onBackClick = null (:81); dialog dismiss only flips showDialog; Confirm launches on the activity-scoped WalletViewModel, so navigating away via the still-enabled drawer icon doesn't cancel the wipe; backgrounding pauses collection but the wipe continues on viewModelScope, and ON_STOP's stopDebounced() is a no-op against an already-stopped node; on failure isWiping clears, a toast fires, and the wallet is intact.
Ordering and lifecycle: remote cleanup doesn't depend on the LDK node, so moving it after stop() is safe, and it runs before keychain.wipe() so the Paykit session is still present — the ordering piotr asked for. stop() can't fail merely because the node never started (LightningService.stop() returns early on node == null), so Forgot-PIN before node start and ErrorStarting still reset. Observer restart on stop() failure runs after setWiping(false), so the new _isWiping guard in startObservingBackups doesn't short-circuit it.
Concurrency: wipeMutex.tryLock + finally unlock is correct, and a second concurrent wipe gets WipeAlreadyInProgress. step() uses runSuspendCatching, so a genuine CancellationException propagates out of the wipe. Cancelling during stopNode/cleanupRemote leaves the wallet intact — stop() is NonCancellable inside its lock and nothing destructive has run. An ON_START node restart mid-wipe is self-healing: wipeStorage's own stop() serialises on lifecycleMutex.
Key material: nothing in the touched code logs a mnemonic or passphrase; VssBackupClient logs only the VSS/LNURL URLs. Keychain.wipe() clears the DataStore and resets the keystore key, and the PR adds no seed-derived artifact that survives it — except the LDK-directory case below. Remote VSS state is intentionally kept for restore; unchanged.
Upgrade: no persisted format changed, and a v2.4.1 VSS backup is read by unchanged BackupRepo code.
I also checked the claim that setup() does no network I/O and it holds — VssClient::new_with_lnurl_auth only derives xprivs and builds the header provider; the JWT is fetched lazily on first request. That's what makes finding 1 below a millisecond-wide window rather than a 30-second one.
jvsena42
left a comment
There was a problem hiding this comment.
ee283d43f closes both LOW observations from my last pass. No blockers.
I verified the wipe fix by reverting the three main-code files to 43449afd6 while keeping the new tests: exactly one failure, invoke should fail before wiping local state when LDK storage wipe fails. So the test is load-bearing rather than decorative, and it asserts verify(keychain, never()).wipe() and verify(db, never()).clearAllTables() rather than just a Result. At head, WipeWalletUseCaseTest 11/11, VssBackupClientTest 6/6, VssBackupClientLdkTest 1/1 all pass. Run in a throwaway worktree outside the repo; nothing committed, working tree untouched.
One consequence of the change I asked for is worth your attention before merge — the abort now lands after cleanupRemote(), so a recoverable "wallet alive, Paykit endpoints unpublished" state is newly reachable. Details and the recovery path are in my reply on the WipeWalletUseCase.kt:88 thread. I'm not asking you to change it; it's a better trade than the leftover-LDK-directory outcome it replaces.
Also checked on the new control flow:
runSuspendCatchingat :59 replacingstopNode().map {}keeps cancellation propagating, andfinallystill clearsisWipingand unlocks. A side benefit: the non-stepcalls inwipeLocal(theresetState()s,resetWalletState(),onSuccess()) now surface as a Result failure and a toast instead of an uncaught exception inviewModelScope.keychain.wipe()is unreachable on the abort path by construction.- A second concurrent wipe still returns
WipeAlreadyInProgress. - The new abort path leaves the user on ResetAndRestore with Back and Reset re-enabled and a toast — it doesn't wedge
isWiping. - Re-walked all seven exits from
ResetAndRestoreScreen(the file didn't change, but the wipe's control flow did): Back while wiping, top-bar back, dialog dismiss, Confirm, drawer navigation mid-wipe, backgrounding, process death before and after the keychain step. All still correct. - No seed-derived material reaches a log or a toast on any new path — the new lines add no
Loggercalls, andNodeStillRunning's message is a fixed string.
On the VSS gate: the capture is right in both clients, and isSetup === gate stops a stale setup clobbering a fresh deferred. I traced the setupWithRetry path specifically for a poisoned-gate regression and there isn't one — an exceptionally-completed deferred reports isCancelled, so the guard doesn't short-circuit and the next setup() gets a fresh gate. Neither client has a test covering that race in either direction; noting it, not asking for one.
jvsena42
left a comment
There was a problem hiding this comment.
ed82850af is correct. No blockers, nothing new to raise as its own thread — one LOW refinement is on the WipeWalletUseCase.kt:88 thread, and it's a consequence of the change I asked for rather than a defect you introduced.
This commit reaches into LightningRepo, which the PR had deliberately left alone, so I reviewed it as a change to shipped lifecycle code rather than as a one-line follow-up. The question that mattered was whether making the early return conditional regresses stop() on the hot path. It doesn't:
- "Stopped with a live node" is not a routine state. The only writer of a non-null node is
setup()understart()'s mutex, and it's reachable only wheninitialLifecycleStatewasStopped— bounded to the 2s retry window, plus a recovery-mode edge.InitializingandErrorStartingalready took the real path before this commit, so a real stop against a set-up-but-unstarted node was already exercised (restartWithElectrumServerfailure →restartWithPreviousConfig). stopDebounced's 5s delay outlives the 2s retry, so a normal background cycle sees no new behaviour. In recovery mode the change is strictly better:start()returns early without touching state, soStopped + live nodeused to persist and make every Recovery-screen wipe abort.- No caller inherits a new failure.
LightningService.stop()can't realistically throw — it'sNonCancellable,node.stop()is insiderunSuspendCatching, andreleaseHandlecatchesdestroy()failures. I walked all tenstop()call sites anyway; the ones that ignore the result (LightningNodeService×2,WakeNodeWorker,onProceedWithoutRestore) get the intended outcome, and the ones that handle failure already did. - The
noderead is safely published:@VolatileatLightningService.kt:171, and both the write at :209 and the read at :618 are underlifecycleMutexregardless. - The fall-through emits
Stopping→Stoppedand replaces the wholeLightningState, which the early return didn't. Every real stop already does this and observers tolerate it; the retry re-derivesisGeoBlocked. No probe-cache leak, since a node that never ran has emitted no events.
Tests are load-bearing: reverting line 618 alone fails exactly stop tears down a node object left alive by a failed start with WantedButNotInvoked: lightningService.stop(). The second case passes in both states, which is right — it pins the no-op branch. 115/115 at head. Throwaway worktree outside the repo, nothing committed, main checkout untouched.
Also confirmed the previous commit's work is byte-identical — the captured VSS setup gate and the fatal LDK-wipe step are untouched, and git diff --stat ee283d43f ed82850af is just these two files.
1a0bce1 to
effa097
Compare
piotr-iohk
left a comment
There was a problem hiding this comment.
Re-reviewed 43449af..effa097 from the diff — not run on device. Both items from my last pass are fixed, and the phased design plus the stop()/lock hardening from the jvsena42 threads look right. One consequence of the phases, fine to take as a follow-up:
- [gap]
keychain.wipe()is now a best-effortstep(). If it throws,wipeLocal()still returns success,resetWalletState()andonSuccess()run, andsetWalletExistsState()sees the mnemonic still there →walletExists = true. The user lands back in a wallet whose LDK dir, Core data, Room and settings are gone but seed and PIN remain, with only awarnlog; before the rewrite this aborted with a toast. Rare (DataStore/Keystore write failure) and recoverable (LDK state rebuilds from VSS, a second Reset retries), so not a merge blocker — but either makekeychain.wipe()a second abort point (fail theResultso the toast fires) or toast when any local step fails. PastwipeStorage, "success" should mean onboarding.
@piotr-iohk Taken in 38c9df8, your first option: |
piotr-iohk
left a comment
There was a problem hiding this comment.
QA from the diff plus emulator (emulator-5554, CI APK 38c9df8). Happy-path Reset and New Wallet backups pass; Pubky cleanup ran with a session before the keychain wipe.
- [nit] If
keychain.wipe()throws,resetWalletState()still sends the user to onboarding while the mnemonic is on the keychain, and the toast says “please reset again” from a screen that no longer has Reset. Rare, recoverable. Not a merge blocker.
On device:
- Reset while the node was Running landed on TOS with the expected order (observers stopped → VSS reset → node stopped → remote cleanup → LDK wiped → keychain wiped). New Wallet then set up VSS on attempt 1 and uploaded METADATA / WIDGETS / ACTIVITY. No
MnemonicNotAvailableException. - Re-tested on a wallet with Pubky + a contact: private Paykit cleanup warned
PrivateUnavailable(best-effort); public cleanup then issueddelete_payment_endpoint(btc-regtest-p2wpkh) anddelete_paykit_receiver_markerbefore keychain wipe. Noidentity_error/no Pubky session. Did not confirm from a second device that the old identity is unresolvable. - Reset-during-
Building node…and the double-tap guard were not reproduced (wipe ~0.5s without Paykit; ~6s with).
jvsena42
left a comment
There was a problem hiding this comment.
Reviewed effa097b7 and 38c9df844. No HIGH/MEDIUM. Two LOWs: one inline below, and one as a reply on the WipeWalletUseCase.kt:88 thread, because it refines the lock I asked for there.
Checked and clean
stop()/stopLocked()split. The body is equivalent:return Result.success(Unit)replacesreturn@withLock, andrunCatching/NonCancellable/onFailureare unchanged.wipeStorageis stillwithContext(bgDispatcher), sostopLockedruns on the same dispatcher.- No re-entrant lock. Nothing inside
wipeStorage's hold acquireslifecycleMutexagain.LightningService.stop()→listenerJob?.cancelAndJoin()cancels any handler waiting on the mutex, sinceMutex.lock()is cancellable.releaseHandle/awaitNodeReleasejoin anioDispatcherjob that only callsdestroy().clearProbeOutcomesandsetRecoveryModeare map and StateFlow writes. All sixstop()call sites enter through the publicstop(). - Hold duration. The lock is now also held across
awaitNodeRelease()(≤90s) and the delete. The callers that can block arestart(),stopDebounced,restartWithPreviousConfigandWakeNodeWorker. The worker's 2-minutewithTimeoutcovers the deliver signal, notstart, so a long hold delays it without failing it. Every entry point is onbgDispatcher, so there is no main-thread wait. - Second Reset after
WipeIncomplete.deleteRecursively()on a missing directory returns true,awaitNodeReleaseis a no-op on a null job, andcleanupRemotewithout a session is swallowed bystep(). A retry from inside the wallet works. - Cancellation.
step()still usesrunSuspendCatching. A cancellation thatstopLocked().mapCatchingwraps insidewipeStorageis rethrown bygetOrThrow()inside the outerrunSuspendCatching. Nothing local is destroyed beforedeleteRecursively. Keychain.wipe()partial failure.edit { clear() }is atomic. If it succeeds andresetEncryptionKey()throws, the entries are gone and the toast is misleading but harmless.- Callers of the new failure.
WalletRepo.wipeWalletis the only caller of the use case.WalletViewModel,RecoveryViewModeland ForgotPin show the failure as a toast;DevSettingsViewModelignores it. Only the use case callslightningRepo.wipeStorage. - Tests.
wipeStorage holds the lifecycle lock…fails without the lock.invoke should fail after wiping the rest when keychain wipe failspins theWipeIncompleteresult. See the inline comment for the landing it also pins. - Logs. No new log line touches seed material, and the
WipeIncompletemessage is a fixed string.
jvsena42
left a comment
There was a problem hiding this comment.
Reviewed 8acaa6759. No HIGH/MEDIUM, one LOW inline. Both fixes from my last pass hold.
Checked and clean
- Wipe flag. It is checked inside
lifecycleMutex(LightningRepo.kt:347) beforeinitialLifecycleStateis read and beforeStartingis emitted. A refused start leaves the lifecycle state alone and schedules no retry, and a start that queued duringwipeStorage's hold is now refused when it takes the lock. The flag is set right afterwipeMutex.tryLock()and cleared infinally, so a failed or cancelled wipe cannot leave it stuck. A start afterfinallyeither finds an empty keychain (MnemonicNotFound) or, onWipeIncomplete, a wallet the user is intentionally kept in. - Callers of the refused start.
WakeNodeWorker.kt:85ignores theResultand waits on the deliver signal as before.LightningNodeService.setupServiceonly acts on success.AppViewModel.completeRNRemoteBackupRestorelogs the error. In recovery mode,RecoveryModeErrorstill wins because it is checked first. WipeIncompletelanding.WalletRepo.wipeWalletreadskeychain.exists(BIP39_MNEMONIC)again afterresetState(). A failedclear()keeps the user in the wallet with Reset available; a failedresetEncryptionKey()alone still lands on onboarding, where Create works.WalletRepoTest.ktpins the first case.- Tests.
start is refused while a wipe is in progresschecks both the failure type and thatlightningService.startis never called. The lock test keepingtimes(2)is correct, because it never sets the flag.
jvsena42
left a comment
There was a problem hiding this comment.
Reviewed bf677b369. No findings, and the toast LOW from my last pass is fixed.
Checked and clean
WalletViewModel.startNode(:344-351) returns early for bothRecoveryModeErrorandWipeInProgressError, logging at debug with no toast. Every other failure still logs at error and shows a toast, same as before.- The only change in behaviour for
RecoveryModeErroris that it is now logged at debug instead of error. Recovery mode is a deliberate user state, so that fits. Logger.debughas noThrowableparameter (Logger.kt:81), so writing the message into the log text is the only way to keep the reason.- No other start caller surfaces
WipeInProgressErrorto the user:WakeNodeWorkerignores theResult,LightningNodeService.setupServiceonly acts on success, andAppViewModel.completeRNRemoteBackupRestoreonly logs. - No existing test asserts on the
RecoveryModeErrortoast branch, so nothing depends on the old logging.
At bf677b369 I have nothing further on this PR. CI was still pending when I checked.
piotr-iohk
left a comment
There was a problem hiding this comment.
QA from the diff plus emulator (emulator-5554, CI APK bf677b3). No new findings. The keychain-incomplete and start-during-wipe follow-ups look right in the diff.
On device:
- Reset while the node was Running landed on TOS with the expected order (observers stopped → VSS reset → node stopped → remote cleanup → LDK wiped → keychain wiped). New Wallet then set up VSS on attempt 1 and uploaded METADATA / ACTIVITY. No
MnemonicNotAvailableException. - Background + resume during Reset did not hit
WipeInProgress— wipe was ~0.6s on this empty wallet. That path is unit-covered only. - Did not re-run the Pubky endpoint delete (last pass on
38c9df8).
* fix: vss client error recovery * fix: stop node before wiping keychain on reset * fix: harden wallet wipe and ldk vss client recovery * fix: wipe wallet in phases so reset never half-completes * fix: gate wipe on ldk storage and capture vss setup deferred * fix: stop a node object left alive by a failed start * fix: hold lifecycle lock across node stop and storage wipe * fix: fail reset when keychain wipe does not complete * fix: refuse node start during wipe and keep wallet on incomplete reset * fix: skip toast for node start refused during wipe
Fixes #1256
Fixes #1257
This PR:
Description
LightningRepo.stop()really stop a node object that a failed start left alive instead of short-circuiting on the Stopped state, so Reset during the start retry window tears the node down before any cleanup rather than failing later at the storage wipe. The storage wipe now holds the lifecycle lock across the stop and the directory removal, and any node start arriving during the wipe is refused under that lock instead of queued, so nothing can rebuild the old seed's node before the keychain is gone.Out of Scope
WalletViewModel.start(): a restore-triggered start dropped by a staleisStartingflag (#1257, remaining part).BackupReporestore picker: falling back to the RN backup when the VSS lookup returns null (#1254).Design
ResetAndRestoreScreengains a loading state on the Reset button while wiping. N/A — no design available.Preview
N/A
QA Notes
Journeys
N/A — no backup/restore journey exists yet.
Manual Tests
Backup succeededin logs).The failed-setup path itself could not be reproduced on-device: setup does no network I/O and the app skips node start while offline, so the only real trigger is the wipe race from #1254. That path is proven by the unit tests only.
Automated Checks
VssBackupClientTest.kt, the same recovery inVssBackupClientLdkTest.kt, and observers being skipped while wiping inBackupRepoTest.kt.LightningRepoTest.kt; wallet existence re-read after an incomplete wipe inWalletRepoTest.kt.WipeWalletUseCaseTest.kt.just compile,just test, andjust lintpass.