fix: stop the test suite wiping the simulator's wallet - #757
Conversation
|
jvsena42
left a comment
There was a problem hiding this comment.
One MEDIUM (test-only): the destructive RN cleanup suite now runs in no lane. Details inline.
Checked and clean:
Env.isUnitTestisXCTestConfigurationFilePath != nilin astatic let. It is never true in Release/TestFlight, in a plain Debug run, in theXCUIApplication-launched app used by E2E and the Trezor UI tests, or inBitkitNotification, so no shipped path reads or writes prefixed accounts.- Every
Keychainentry point (save,update,delete,upsert,load, plus the helpers that delegate to them) goes throughaccount(for:). The directSecItem*calls inMigrationsServicekey RN items by service andTrezorCredentialStorageuses its own service, so neither is inconsistent with the prefix. InstallationMarker: when not under test the path is unchangedDocuments, andcreateDirectory(withIntermediateDirectories:)is a no-op there. ThehandleOrphanedKeychainlogic is untouched. The marker-deleting suites now only touchDocuments/unit-tests/.RNMigrationCleanupTeststhrows its skip before any deletion, andtearDownis gated ondidRunDestructiveSetUp, so the skipped path runs nothing destructive.- The
-only-testing/-skip-testingnames match real classes.TxBumpingTestsno longer exists. KeychainTestsasserts throughKeychain.account(for:)and the access group, so the tests are not vacuous. TheChannelMigrationPersistenceTestssnapshot restoresrnPendingChannelMigrationin a teardown block.
5e143a4 to
20f9f8f
Compare
20f9f8f to
7019cb6
Compare
7019cb6 to
0942385
Compare
There was a problem hiding this comment.
Verdict: ✅ Approve
Reaudit: diff 2 files.
Findings:
N/A
Audit:
Audited - no findings.
Coverage:
QA: Manual Tests await all reviewers to approve, author can run it now via comment: @ovi-reviewer test
Reviewed by gpt-6-astra-xhigh via gh-pr-review-loop skill
Commands: @ovi-reviewer test · retest · audit (author or owner)
|
QA reviewed on cc4ff06. Reviewed the keychain account prefix, installation-marker redirect, RN cleanup gate, and the two tests that fail if those namespaces are removed; no device run this round, test-only / no UI. No findings. Checked and clean
QA LGTM |
`BitkitTests` is hosted in the Bitkit app and `Env.network` resolves to regtest for both a test run and a Debug build, so they share the access group `KYH47R284B.to.bitkit.regtest`. Eight suites call `wipeEntireKeychain()` from setUp or tearDown, unguarded — four of them in the default unit lane — so running the suite deletes the developer's seed, PIN and Pubky/Paykit identity. Give entries a `unit-tests.` account prefix when `XCTestConfigurationFilePath` is set, and make the wipe skip anything un-prefixed while under test. This is the lever `Env.appStorageUrl` already uses to keep LDK and bitkit-core storage out of the app's own directories; it needs no entitlement change, leaves kSecAttrAccessible, kSecAttrSynchronizable and the access-group pinning untouched, and only ever narrows what a process can reach. Deliberately not a compile-time flag: `UNIT_TESTING` is defined for the test target only, so `Bitkit.Keychain` — the copy `StartupHandler` and the services under test actually call — would keep writing real accounts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcCTBgiMafXxw71MWBGB2T
InstallationMarkerTests and OrphanedKeychainTests delete the marker in tearDown, and it lives in the real sandbox Documents directory — deliberately outside the app group, so it is not covered by the `Env.appStorageUrl` redirect. A missing marker is how the app detects a keychain orphaned by a reinstall: `AppScene.handleOrphanedKeychain` sees no marker, finds a mnemonic, and wipes the keychain on the next launch. So a test run arms a wallet wipe that fires later, and namespacing the keychain alone would have made that worse — the wallet would survive the run only to be deleted when the app next started. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcCTBgiMafXxw71MWBGB2T
`MigrationsService.pendingChannelMigration` is backed by UserDefaults.standard, so setUp and tearDown were nilling the host app's own key. On a device mid- migration from React-Native Bitkit that payload is the channel manager and its monitors. Snapshot the key and let the restore put it back, rather than nilling it again in tearDown — the previous value is what the suite should leave behind, not nil. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcCTBgiMafXxw71MWBGB2T
Unlike the native keychain, this suite cannot be namespaced away: it deletes the real ~/Documents/mmkv and ~/Documents/ldk, and `cleanupRNKeychain` deletes RN keychain items by service with no access-group filter. On a device mid-migration from React-Native Bitkit, that is the migration source. Skip it unless BITKIT_DESTRUCTIVE_TESTS=1, and set that only in the integration lane, which already erases its simulator between attempts. Move the suite into that lane's -only-testing list so it keeps running somewhere rather than being silently dropped, and out of the unit lane. Also drops -skip-testing:BitkitTests/TxBumpingTests, which names a class that no longer exists. The gate guards tearDown too: XCTest runs it even when setUp skips, and the cleanup is the destructive part. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcCTBgiMafXxw71MWBGB2T
The gate never received it. xcodebuild does not pass the invoking shell's environment to the simulator-hosted test runner — only variables named TEST_RUNNER_<VAR>, forwarded with the prefix stripped. So the guard read a variable that never arrived, all twelve tests in the suite self-skipped in the integration lane as well as the unit lane, and because skips do not fail a lane both workflows stayed green with the suite running nowhere. That is the outcome moving the suite into the integration lane was meant to prevent. Verified both halves on a simulator: BITKIT_DESTRUCTIVE_TESTS=1 xcodebuild test … 1 test, 1 skipped TEST_RUNNER_BITKIT_DESTRUCTIVE_TESTS=1 xcodebuild test … 1 test, 0 skipped The guard keeps reading the unprefixed name, which is what the runner sees. The helper's documentation now says how to set it locally, since the obvious spelling fails silently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcCTBgiMafXxw71MWBGB2T
…espace
`KeychainTests` only asserted that entries saved through `account(for:)` were gone
after a wipe, so it passed with or without the guard that keeps the wipe inside the
unit-test namespace. Removing the guard would wipe a simulator wallet again while
the suite stayed green.
Plant an account outside the namespace, wipe, and assert it survives. The probe
uses a unique throwaway name rather than a real key like `bip39_mnemonic_0`: on a
simulator with a wallet that account already exists, and deleting it by name in
teardown would remove the real seed.
Verified on an erased simulator: passes with the guard, fails with it removed
("wipeEntireKeychain deleted an account outside the unit-test namespace").
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AcCTBgiMafXxw71MWBGB2T
`testMarkerPathUsesSandboxDocuments` only checked that the marker sits under Documents, which the real path already did, so removing the `unit-tests` redirect would let `delete()` remove the real marker while every test passed. Assert the path is inside `unit-tests`, and that the real `Documents/.bitkit_installed` survives `create()` and `delete()`. The stand-in is planted only when no real marker exists and only that stand-in is removed — on a simulator with a real install, deleting that file makes the next launch wipe the keychain. Verified on an erased simulator: both pass with the redirect and fail without it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcCTBgiMafXxw71MWBGB2T
cc4ff06 to
a8f95e5
Compare
This PR stops the unit test suite destroying the wallet on the simulator it runs against.
Running the suite deletes the seed, the PIN and the whole Pubky and Paykit identity: the app reopens at Terms of Use and the regtest ledger on disk is orphaned, because without the mnemonic the app will not adopt it. Eight suites call the keychain wipe from setUp or tearDown with no guard, and four of those are in the default unit lane rather than the integration one. The test bundle is hosted in the app, and the network resolves to regtest for both a test run and a Debug build, so they share one keychain access group.
Entries now carry a test-only account prefix when the process is running tests, and the wipe skips anything un-prefixed while under test. This is the same lever the app already uses to keep its LDK and core storage out of the real directories, it needs no entitlement change, and it leaves the accessibility, synchronisability and access-group pinning alone. It only ever narrows what a process can reach. A compile-time flag would not have worked here: the flag is defined for the test target only, so the copy of the keychain helper that the services under test actually call would have kept writing real accounts.
The installation marker has to move with it, and this is the part worth a second look. Two suites delete the marker, which lives in the real sandbox Documents directory rather than the app group, so it is outside the existing redirect. A missing marker is how the app detects a keychain orphaned by a reinstall: on the next launch it finds no marker, finds a mnemonic, and wipes the keychain. So a test run arms a wipe that fires later. Today that is invisible because the keychain is already empty by then — fixing the keychain alone would have made it worse, with the wallet surviving the run only to be deleted when the app next started.
Two other things came out of auditing the same area. The channel migration tests were nilling a real pending migration, which is a React-Native channel manager and its monitors, so that key is now snapshotted and restored. And the React-Native cleanup suite cannot be namespaced away at all — it deletes the real
mmkvandldkdirectories under Documents and removes keychain items by service with no access-group filter, which on a device mid-migration is the migration source. That one is now opt-in behind an environment variable, set only in the integration lane, which already erases its simulator between attempts. It moves into that lane's test list so it keeps running somewhere rather than being quietly dropped.Also drops a skip for a test class that no longer exists.
No changelog fragment: the namespacing is inert unless the process is running tests, and no shipped behaviour changes.
Linked Issues/Tasks
Follows #756. Found while auditing the suite for #733.
Design
N/A — no UI changes.
QA Notes
Manual Tests
regression:Settings → Security: PIN and biometrics settings are as they were before the run.Automated Checks
BitkitTests/KeychainTests.swiftaddresses entries through the same helper the app uses, since it builds its own keychain queries.BitkitTests/RNMigrationCleanupTests.swiftskips unlessBITKIT_DESTRUCTIVE_TESTS=1; the guard covers tearDown too, because XCTest runs it even when setUp skips and the cleanup is the destructive part.