Fix the Android Zano freeze, deadlock, and file recovery - #20
Conversation
j0ntz
left a comment
There was a problem hiding this comment.
Neither of these blocks the PR. RnMoneroModule in react-native-monero-lwsf uses the identical executor pattern, so both apply there too.
| // scan chunk, or the SDK's close-during-scan lock inversion) froze every | ||
| // view update in the app. A single thread preserves the strict global | ||
| // call ordering the shared bridge thread provided. | ||
| private final ExecutorService executor = Executors.newSingleThreadExecutor(); |
There was a problem hiding this comment.
Warning: scoping the executor to the module instance narrows the ordering guarantee the comment above claims.
mqt_native_modules is one thread per process, so its ordering held across React context recreations. An instance field does not. A JS reload builds a new RnZanoModule with a second executor while the first one's thread is still alive and possibly still draining queued calls, so two threads then call into the same process-global Zano SDK. That is the concurrent close-during-scan shape this PR exists to contain.
sequenceDiagram
participant JS
participant Exec1 as executor, instance 1
participant Exec2 as executor, instance 2
participant SDK as Zano SDK, process-global
JS->>Exec1: closeWallet(w)
Note over Exec1,SDK: blocked in C++
JS->>JS: reload rebuilds the ReactContext
JS->>Exec2: open(w)
Exec2->>SDK: open
Exec1->>SDK: closeWallet resumes
Note over SDK: two threads, ordering gone
private static final ExecutorService executor restores the process-global property and drops the per-reload thread leak in the same line. An invalidate() override calling shutdown() fixes only the leak: the in-flight call still races the new instance.
Same declaration, separate nit: Executors.newSingleThreadExecutor(r -> new Thread(r, "zano")) names the thread. The evidence in this PR's description is a thread dump, and pool-1-thread-1 will not identify itself in the next one.
There was a problem hiding this comment.
Both taken, in ca1dc61: the executor is now static final, and the thread is named "zano" so it identifies itself in the next dump.
One sharpening on the mechanism from re-checking RN's source: mqt_native_modules is per-CatalystInstance rather than per-process, so the pre-PR ordering did not actually survive reloads either. That makes the static executor a strengthening rather than a restoration -- and your scenario is the one that makes it load-bearing rather than hygiene: with the close_wallet patch in this PR, a mid-close wallet is detached from the manager's map before its store lands, so a reload-spawned second thread could double-open the very file the old thread is still writing. The instance-scoped executor was the only thing standing between those two facts.
The same pair of fixes for RnMoneroModule in react-native-monero-lwsf is queued as a follow-up, per your note.
| executor.execute(() -> { | ||
| try { | ||
| promise.resolve(callZanoJNI(method, strings)); | ||
| } catch (Exception e) { |
There was a problem hiding this comment.
Suggestion: catch (Exception e) no longer covers everything the promise depends on.
The clause is unchanged, but the thread under it is not. On mqt_native_modules an escaping Error reached RN's NativeModuleCallExceptionHandler (redbox in dev, an RN-owned crash in prod). Out of an executor.execute Runnable it goes to the default uncaught handler instead, which kills the process. promise was never settled in that path before either; catch (Throwable e) costs one word and gets the JS caller a ZanoError instead of silence.
There was a problem hiding this comment.
Done in ca1dc61 -- catch (Throwable), with a comment carrying the rationale: off the bridge thread an escaping Error reaches the default uncaught handler and kills the process with the promise unsettled, where RN's own dispatcher used to own it.
On the legacy Android architecture every @ReactMethod runs on the single shared mqt_native_modules thread, which also executes UIManager's view commands, so a Zano call that blocked in C++ froze every view update in the app: taps dead, screens frozen, native scrolling alive. A JDWP dump of the frozen app shows callZano parked on that thread inside callZanoJNI. callZano now hops to a dedicated single-thread executor and settles the promise from there. The executor is static: the Zano SDK is one instance per process while React module instances are not, so a JS reload would otherwise run the old instance's still-draining call concurrently with the new instance's calls. A single process-wide thread preserves the strict global call ordering across reloads too, and is named "zano" so it identifies itself in future dumps. The catch widens to Throwable: off the bridge thread an escaping Error would hit the default uncaught handler and kill the process with the promise unsettled, where RN's dispatcher previously caught it. Argument extraction from the ReadableArray stays on the caller thread.
bc65a54 to
40c3ad5
Compare
j0ntz
left a comment
There was a problem hiding this comment.
Read the executor change and the INVALID_FILE recovery too, and both look right to me. The notes below are all on the close_wallet patch.
|
|
||
| it->second.major_stop = true; | ||
| it->second.stop_for_refresh = true; | ||
| it->second.w.unlocked_get()->stop(); |
There was a problem hiding this comment.
Warning: stop() moved out of the try/catch, so its failures change shape at the JS boundary.
In the pinned original this call sat inside the try alongside store() and the erase, so a throw came back as API_RETURN_CODE_FAIL:<what> or API_RETURN_CODE_INTERNAL_ERROR. Here it runs in the lock scope that closes on line 117, before the try opens on line 119. A throw now unwinds out of close_wallet (the RAII lock still releases, so no deadlock) into the catch-all at src/jni/jni.cpp:47-59, which ThrowNews a Java exception. That reaches JS as a rejected promise rather than a resolved { response: 'FAIL:...' }.
Every caller is written against the return-code shape: CppBridge.ts:417 decides from closeResponse whether deleting the file is safe, and lines 596 and 624 branch on response !== 'OK'. It also contradicts the docstring above, which says the deliberate semantics changes are "all confined to the window after the node leaves the map" (line 71). This one happens before the extract.
Opening the try before the lock scope, with EXCLUSIVE_CRITICAL_REGION_LOCAL inside it, restores the original contract without touching the deadlock fix. Nothing in test/closeWalletPatch.test.ts pins it either way.
There was a problem hiding this comment.
Fixed in ea9638d (now 67a8fce after the rebase): the try opens before the lock scope, so stop(), the extract and the find are back inside it and failures keep reporting as return codes. The node stays declared outside the try, so the worker join in its destructor still runs at function exit rather than during unwinding -- a join() throwing mid-unwind would be a std::terminate the original could not produce.
You are right that the docstring was wrong as written; it now says the changes are confined to the window after the extract, which is true again. Added a test that walks the patched body and asserts every throwing step sits after the try and the node declaration sits before it.
| * leaves the map: | ||
| * | ||
| * - "Absent from `m_wallets`" no longer implies "closed and stored". | ||
| * During the store/join window the wallet is invisible to |
There was a problem hiding this comment.
Question: does this list need a third bullet for the refresh worker's own callbacks?
The deadlock analysis above names on_transfer2 and on_transfer_canceled as the path by which the refresh worker re-enters the manager and takes m_wallets_lock shared. Those callbacks fire during exactly the store/join window this bullet describes, when the node is out of m_wallets, so a lookup by wallet_id there now misses where it previously found the entry. The bullet covers open_wallet's ALREADY_EXISTS check and status calls, but not the one caller guaranteed to be running.
I do not have the SDK sources checked out to see what that path does with a miss. A dropped notification during a close is harmless; an unchecked ->second on end() is not. If it is safe, saying so in this list would settle it for the next reader and for the upstream submission this docstring is already drafting.
There was a problem hiding this comment.
Checked it against the pinned sources: safe, and now written into the list as a third bullet.
on_transfer2 goes through GET_WALLET_OPTIONS_BY_ID_VOID_RET (wallets_manager.cpp:37-42), which does a checked find and returns on a miss; on_transfer_canceled has its own end() test that logs and returns. So a callback landing in the store/join window drops a view notification for a wallet that is closing anyway. Every m_wallets.find in the file is checked the same way, and the three bare m_wallets[...] sites all insert a freshly counted id in the open, restore and generate paths -- ids come from a monotonic counter, so a late callback cannot hit a recycled one either.
| function normalize(code: string): string { | ||
| return code | ||
| .split('\n') | ||
| .map(line => line.replace(/[ \t]+$/, '')) |
There was a problem hiding this comment.
Suggestion: normalize does not strip \r.
The regex runs per line after split('\n'), so a CRLF checkout leaves a trailing \r on every line, and the byte-for-byte comparison then fails against the LF-only original on unmodified upstream source. The build stops with a message pointing at a drift that did not happen. /[ \t\r]+$/ covers it.
Low odds on macOS and Linux defaults, and zano_native_lib may well ship a .gitattributes, so this is a one-character hedge rather than a live bug.
There was a problem hiding this comment.
Fixed -- the class is /[ \t\r]+$/ now, with a test that feeds the transform a CRLF copy of the pinned source and expects it to patch rather than report drift.
The SDK's close_wallet holds m_wallets_lock exclusively across two waits on the wallet being closed: the store() call, which needs the per-wallet lock the refresh worker holds for the whole of a scan chunk, and the map erase, whose destructor joins the worker thread. The refresh path re-enters the manager through wallet callbacks that take m_wallets_lock shared, so a close issued while the wallet is catching up wedges the worker, the close, and then every Zano call in the process, permanently. The app's periodic mid-sync checkpoint saves trip this within minutes of importing a wallet on Android. update-sources now rewrites close_wallet in the downloaded sources to detach the map node under the lock and do the store and the join with no manager lock held. Everything that can throw stays inside the try, so failures keep reporting as return codes rather than escaping into the JNI catch-all, which callers do not expect; the node is declared outside it so the worker join in its destructor runs at function exit rather than during unwinding. The log-prefix write gains a bounds check: the manager lock no longer serializes it against reset()'s vector clear, so the blind vector[wallet_id] store could go out of bounds. The transform requires the found function to match the pinned original exactly, modulo trailing whitespace and line endings, so a pin bump that changes close_wallet in any way fails the build for a human to re-evaluate the patch instead of silently keeping or dropping it. Only Android builds from these sources; iOS links the prebuilt xcframework and keeps the original blocking semantics.
A crash during a wallet file's very first write leaves a file the SDK cannot parse -- typically zero bytes -- and opening it fails with INVALID_FILE before any password is consulted. startWallet's recovery ladder is keyed on WRONG_PASSWORD, so the error propagated as-is and the engine retried the same doomed open every second, forever: a full native init and load per attempt, per wallet, with no path back to health. Observed live after an emulator was hard-killed mid-restore: two of four wallets left zero-byte files and spun in the retry loop while the other two synced. Route INVALID_FILE to the same policy as a file no known password opens, skipping the pointless password ladder: with no seed passphrase set, delete the file and rebuild it from the mnemonic, which recreates the identical wallet at the cost of a re-scan; with a passphrase set, refuse with a clear error, since an unreadable file cannot corroborate the passphrase and a wrong one would rebuild a different wallet.
c64eece to
67a8fce
Compare
CHANGELOG
Does this branch warrant an entry to the CHANGELOG?
Dependencies
none — supersedes EdgeApp/edge-currency-accountbased#1094: with the deadlock
fixed at its source here, accountbased needs no changes at all, and
checkpointing stays enabled with identical code on both platforms.
Description
Fixes the Android freeze QA reported — about seven minutes after login, taps
and buttons die while scrolling and the drawer keep working, and the Zano
sync banner stops updating; iOS is unaffected — in three commits: one
contains the blast radius of any blocked Zano call, one removes the deadlock
that produced the blocked call, and one fixes a wallet-file recovery gap the
deadlock's verification uncovered.
Commit 1 — keep Zano calls off the RN bridge thread.
On the legacy Android architecture every
@ReactMethodruns on the singleshared
mqt_native_modulesthread, andcallZanoran its JNI synchronouslythere. That thread also executes UIManager's view commands, so any Zano call
that blocks in C++ freezes every view update in the app: buttons dead
(UIManager), sync banner frozen (every native call queued behind the blocker,
including the
tryPullResultpolls that would have observed the closefinishing), scroll and drawer alive (pure UI thread). QA's screen recording
matches this signature exactly — the sync banner reads the same block count
across four minutes.
Repro evidence (API 35 emulator, checkpoints tortured to a 30s interval,
4 wallets in deep catch-up): a JDWP thread dump captured mid-freeze shows
callZanonow hops to a dedicated single-thread executor and settles thepromise from there. Argument extraction from the
ReadableArraystays on thecaller thread, and a single thread preserves the strict global call ordering
the shared bridge thread provided. Re-running the identical torture scenario
with this change, the blocked call sits quarantined on the executor thread
while
mqt_native_modules,mqt_js, andmainall stay idle and renderingcontinues.
Per review: the executor is
static— the Zano SDK is one instance perprocess while React module instances are not, so a JS reload would otherwise
run the old instance's still-draining call concurrently with the new
instance's calls (with commit 2, that overlap could double-open a wallet
file mid-store). The thread is named
"zano"so it identifies itself infuture dumps, and the catch widens to
Throwable, since off the bridgethread an escaping
Errorwould kill the process with the promiseunsettled. The change needs no architecture detection: it never blocks the
thread that called it, which is correct under both the legacy bridge and
the new-arch interop dispatch.
Commit 2 — patch the SDK's close-during-scan deadlock.
The call that blocked is
close_wallet, issued by the engine's mid-synccheckpoint save. The pinned SDK implementation holds
m_wallets_lockexclusively across two waits on the wallet being closed: the
store()call,which needs the per-wallet lock the refresh worker holds for the whole of a
scan chunk, and the map erase, whose destructor joins the worker thread. The
refresh path re-enters the manager through wallet callbacks that take
m_wallets_lockshared (the SDK documents the ordering hazard in itson_sync_progresscomment), and every other API call takes it shared upfront — so one close issued mid-catch-up wedges the worker, the close, and
then every Zano call in the process, permanently. In two of two torture runs
the wedge never cleared and no checkpoint ever landed again.
update-sourcesnow rewritesclose_walletin the downloaded sources:detach the map node while holding the lock, then do the store and the
implicit worker join with no manager lock held. Node extraction keeps the
element at its address, so the worker's references stay valid, and wallet
ids are never reused. The stop flags, the store-then-log-prefix order, and
the return codes are kept verbatim; the log-prefix write gains a bounds
check, since the manager lock no longer serializes it against
reset()'svector clear. The transform requires the found function to match the pinned
original exactly (modulo trailing whitespace), so a pin bump that changes
close_walletin any way fails the build for a human to re-evaluate thepatch instead of silently keeping or dropping it
(
scripts/utils/closeWalletPatch.ts, unit-tested).Two deliberate semantics changes, documented in the transform: during the
store/join window the wallet is absent from the map while its
wallet2still writes the file — safe for this bridge, which strictly sequences
close-before-reopen per wallet on one executor thread, and called out for
any upstream submission. And when
store()throws, the original left theentry in the map as a zombie (stop flags set, never able to sync again),
while the rewrite reports the same error with the wallet gone.
Only Android builds from these sources; iOS links the prebuilt
libzano-plain-walletxcframework and is byte-for-byte unchanged. iOS hasnot exhibited the wedge across hundreds of observed checkpoint cycles,
though the lock inversion exists in its prebuilt code too — the durable fix
for both platforms is handing this patch upstream with the next pin-bump
request.
Commit 3 — rebuild unreadable wallet files.
Found while verifying the deadlock fix: a crash during a wallet file's very
first write leaves a file the SDK cannot parse — typically zero bytes — and
opening it fails with
INVALID_FILEbefore any password is consulted(
wallet2's header read maps toAPI_RETURN_CODE_INVALID_FILE).startWallet's recovery ladder is keyed onWRONG_PASSWORD, so the errorpropagated as-is and the engine retried the same doomed open every second,
forever — a full native init and load per attempt, per wallet, with no path
back to health. Observed live after an emulator was hard-killed mid-restore:
two of four wallets left zero-byte files and spun at 28 failed opens per 20
seconds while the other two synced.
INVALID_FILEnow routes to the same policy as a file no known passwordopens, skipping the pointless password ladder: with no seed passphrase set,
the file is deleted and rebuilt from the mnemonic, recreating the identical
wallet at the cost of a re-scan; with a passphrase set, it refuses with a
clear error, since an unreadable file cannot corroborate the passphrase and
a wrong one would rebuild a different wallet. This half is platform-neutral
JS. On-device: both stuck wallets logged the new path, rebuilt, and synced;
the error spam went to zero.
Verification. Unit suite green (53 tests). A full
update-sourcesrunfrom a fresh checkout applies the patch and rebuilds all four Android ABIs
and the iOS xcframework cleanly, with
wallets_manager.cpprecompiled perABI. The emulator torture repro that wedged 2/2 before the patch — 30-second
checkpoint interval, four wallets in deep catch-up — ran 25 minutes against
the rebuilt library: 41 completed checkpoint cycles, zero stalls, the
app process alive and interactive throughout, with each cycle also
re-opening the file the previous patched close wrote. A later session added
stock-interval (5-minute) checkpoint cycles landing on schedule.