Skip to content

fix: stop the test suite wiping the simulator's wallet - #757

Merged
jvsena42 merged 7 commits into
masterfrom
fix/733-stop-test-suites-wiping-the-wallet
Sep 22, 2026
Merged

jvsena42 merged 7 commits into
masterfrom
fix/733-stop-test-suites-wiping-the-wallet

Conversation

@jvsena42

@jvsena42 jvsena42 commented Sep 16, 2026

Copy link
Copy Markdown
Member

Stack — review in order; each targets the one before it.

  1. fix: keep transfer unit tests out of the app's UserDefaults #756 — keep transfer unit tests out of the app's UserDefaults (fixes fix: hardware transfer unit tests write mock transfers into the app's UserDefaults #733)
  2. fix: stop the test suite wiping the simulator's wallet #757 — stop the test suite wiping the simulator's wallet
  3. test: isolate remaining suites from the app's state #758 — isolate remaining suites from the app's state

This PR stops the unit test suite destroying the wallet on the simulator it runs against.

Running the suite deletes the seed, the PIN and the whole Pubky and Paykit identity: the app reopens at Terms of Use and the regtest ledger on disk is orphaned, because without the mnemonic the app will not adopt it. Eight suites call the keychain wipe from setUp or tearDown with no guard, and four of those are in the default unit lane rather than the integration one. The test bundle is hosted in the app, and the network resolves to regtest for both a test run and a Debug build, so they share one keychain access group.

Entries now carry a test-only account prefix when the process is running tests, and the wipe skips anything un-prefixed while under test. This is the same lever the app already uses to keep its LDK and core storage out of the real directories, it needs no entitlement change, and it leaves the accessibility, synchronisability and access-group pinning alone. It only ever narrows what a process can reach. A compile-time flag would not have worked here: the flag is defined for the test target only, so the copy of the keychain helper that the services under test actually call would have kept writing real accounts.

The installation marker has to move with it, and this is the part worth a second look. Two suites delete the marker, which lives in the real sandbox Documents directory rather than the app group, so it is outside the existing redirect. A missing marker is how the app detects a keychain orphaned by a reinstall: on the next launch it finds no marker, finds a mnemonic, and wipes the keychain. So a test run arms a wipe that fires later. Today that is invisible because the keychain is already empty by then — fixing the keychain alone would have made it worse, with the wallet surviving the run only to be deleted when the app next started.

Two other things came out of auditing the same area. The channel migration tests were nilling a real pending migration, which is a React-Native channel manager and its monitors, so that key is now snapshotted and restored. And the React-Native cleanup suite cannot be namespaced away at all — it deletes the real mmkv and ldk directories under Documents and removes keychain items by service with no access-group filter, which on a device mid-migration is the migration source. That one is now opt-in behind an environment variable, set only in the integration lane, which already erases its simulator between attempts. It moves into that lane's test list so it keeps running somewhere rather than being quietly dropped.

Also drops a skip for a test class that no longer exists.

No changelog fragment: the namespacing is inert unless the process is running tests, and no shipped behaviour changes.

Linked Issues/Tasks

Follows #756. Found while auditing the suite for #733.

Design

N/A — no UI changes.

QA Notes

Manual Tests

  • 1a. On a simulator carrying a wallet and a Pubky identity, run the full unit lane → launch the app: still opens to the wallet, not Terms of Use, with the identity and its contact links intact.
    • 1b. launch a second time: wallet still there. This is the launch where the orphan check would fire if the marker fix were missing.
  • 2. regression: Settings → Security: PIN and biometrics settings are as they were before the run.

Automated Checks

  • Unit tests modified: BitkitTests/KeychainTests.swift addresses entries through the same helper the app uses, since it builds its own keychain queries.
  • Test gating added: BitkitTests/RNMigrationCleanupTests.swift skips unless BITKIT_DESTRUCTIVE_TESTS=1; the guard covers tearDown too, because XCTest runs it even when setUp skips and the cleanup is the destructive part.
  • CI: the integration workflow sets that variable and gains the suite; the unit workflow skips it.
  • Verified locally on iPhone 17 against a live wallet: the suites that previously wiped it now leave it intact, the marker keeps its original timestamp, and the tests create and delete their own marker in a separate directory.

@greptile-apps

greptile-apps Bot commented Sep 16, 2026

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

The PR appears safe to merge; the changed isolation boundaries consistently protect simulator wallet and migration state while retaining destructive coverage in the designated integration lane.

Summary

This PR isolates test-owned keychain entries and installation markers from simulator wallet state, preserves pending channel-migration preferences across tests, and confines destructive React Native migration cleanup to the integration lane.

  • Prefixes native keychain accounts only in hosted test processes and prevents test wipes from deleting unprefixed entries.
  • Redirects the installation marker to a test-only Documents subdirectory.
  • Snapshots and restores the pending channel-migration preference.
  • Gates destructive legacy migration cleanup behind an explicit environment variable and updates CI lane selection.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  U[Unit test process] --> T{Env.isUnitTest}
  T -->|true| K[Test-prefixed keychain accounts]
  T -->|true| M[Documents/unit-tests marker]
  K --> W[Test wipe deletes only prefixed accounts]
  D[RN migration cleanup suite] --> G{BITKIT_DESTRUCTIVE_TESTS=1}
  G -->|false| S[Skip without destructive teardown]
  G -->|true in integration lane| C[Clean legacy mmkv, ldk, and RN keychain data]
Loading

Reviews (1) · Last reviewed commit: "test: gate the RN migration cleanup suit..."

@jvsena42
jvsena42 added this pull request to stack #759 September 16, 2026 14:26
@jvsena42 jvsena42 self-assigned this Sep 16, 2026

@jvsena42 jvsena42 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One MEDIUM (test-only): the destructive RN cleanup suite now runs in no lane. Details inline.

Checked and clean:

  • Env.isUnitTest is XCTestConfigurationFilePath != nil in a static let. It is never true in Release/TestFlight, in a plain Debug run, in the XCUIApplication-launched app used by E2E and the Trezor UI tests, or in BitkitNotification, so no shipped path reads or writes prefixed accounts.
  • Every Keychain entry point (save, update, delete, upsert, load, plus the helpers that delegate to them) goes through account(for:). The direct SecItem* calls in MigrationsService key RN items by service and TrezorCredentialStorage uses its own service, so neither is inconsistent with the prefix.
  • InstallationMarker: when not under test the path is unchanged Documents, and createDirectory(withIntermediateDirectories:) is a no-op there. The handleOrphanedKeychain logic is untouched. The marker-deleting suites now only touch Documents/unit-tests/.
  • RNMigrationCleanupTests throws its skip before any deletion, and tearDown is gated on didRunDestructiveSetUp, so the skipped path runs nothing destructive.
  • The -only-testing/-skip-testing names match real classes. TxBumpingTests no longer exists.
  • KeychainTests asserts through Keychain.account(for:) and the access group, so the tests are not vacuous. The ChannelMigrationPersistenceTests snapshot restores rnPendingChannelMigration in a teardown block.

Comment thread .github/workflows/integration-tests.yml Outdated
Comment thread BitkitTests/AppStateIsolation.swift
@jvsena42
jvsena42 force-pushed the fix/733-stop-test-suites-wiping-the-wallet branch from 5e143a4 to 20f9f8f Compare September 16, 2026 15:12
@jvsena42
jvsena42 force-pushed the fix/733-stop-test-suites-wiping-the-wallet branch from 20f9f8f to 7019cb6 Compare September 17, 2026 10:07
ovi-reviewer[bot]

This comment was marked as resolved.

@jvsena42
jvsena42 force-pushed the fix/733-stop-test-suites-wiping-the-wallet branch from 7019cb6 to 0942385 Compare September 17, 2026 10:55

@ovi-reviewer ovi-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: ✅ Approve


Reaudit: diff 2 files.

Findings:
N/A

Audit:
Audited - no findings.

Coverage:
QA: Manual Tests await all reviewers to approve, author can run it now via comment: @ovi-reviewer test


Reviewed by gpt-6-astra-xhigh via gh-pr-review-loop skill
Commands: @ovi-reviewer test · retest · audit (author or owner)

@piotr-iohk

Copy link
Copy Markdown
Collaborator

QA reviewed on cc4ff06.

Reviewed the keychain account prefix, installation-marker redirect, RN cleanup gate, and the two tests that fail if those namespaces are removed; no device run this round, test-only / no UI.

No findings.

Checked and clean

  • Env.isUnitTest is XCTestConfigurationFilePath; prefixed accounts and Documents/unit-tests only apply in the XCTest-hosted process, not Debug / E2E / notifications.
  • wipeEntireKeychain under test skips un-prefixed accounts; KeychainTests plants a throwaway probe so removing that skip goes red.
  • InstallationMarker.delete() is asserted not to remove Documents/.bitkit_installed; a missing real marker is still what handleOrphanedKeychain uses on a normal launch.
  • RNMigrationCleanupTests is skipped in the unit lane, opted in via TEST_RUNNER_BITKIT_DESTRUCTIVE_TESTS on integration, and tearDown does nothing if setUp skipped.
  • Channel migration tests snapshot rnPendingChannelMigration instead of nilling it.

QA LGTM

@ovitrif ovitrif left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utAck

Base automatically changed from fix/733-transfer-test-userdefaults-pollution to master September 21, 2026 10:28
jvsena42 and others added 7 commits September 21, 2026 07:28
`BitkitTests` is hosted in the Bitkit app and `Env.network` resolves to regtest
for both a test run and a Debug build, so they share the access group
`KYH47R284B.to.bitkit.regtest`. Eight suites call `wipeEntireKeychain()` from
setUp or tearDown, unguarded — four of them in the default unit lane — so running
the suite deletes the developer's seed, PIN and Pubky/Paykit identity.

Give entries a `unit-tests.` account prefix when `XCTestConfigurationFilePath` is
set, and make the wipe skip anything un-prefixed while under test. This is the
lever `Env.appStorageUrl` already uses to keep LDK and bitkit-core storage out of
the app's own directories; it needs no entitlement change, leaves
kSecAttrAccessible, kSecAttrSynchronizable and the access-group pinning untouched,
and only ever narrows what a process can reach.

Deliberately not a compile-time flag: `UNIT_TESTING` is defined for the test
target only, so `Bitkit.Keychain` — the copy `StartupHandler` and the services
under test actually call — would keep writing real accounts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AcCTBgiMafXxw71MWBGB2T
InstallationMarkerTests and OrphanedKeychainTests delete the marker in tearDown,
and it lives in the real sandbox Documents directory — deliberately outside the
app group, so it is not covered by the `Env.appStorageUrl` redirect.

A missing marker is how the app detects a keychain orphaned by a reinstall:
`AppScene.handleOrphanedKeychain` sees no marker, finds a mnemonic, and wipes the
keychain on the next launch. So a test run arms a wallet wipe that fires later,
and namespacing the keychain alone would have made that worse — the wallet would
survive the run only to be deleted when the app next started.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AcCTBgiMafXxw71MWBGB2T
`MigrationsService.pendingChannelMigration` is backed by UserDefaults.standard,
so setUp and tearDown were nilling the host app's own key. On a device mid-
migration from React-Native Bitkit that payload is the channel manager and its
monitors.

Snapshot the key and let the restore put it back, rather than nilling it again in
tearDown — the previous value is what the suite should leave behind, not nil.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AcCTBgiMafXxw71MWBGB2T
Unlike the native keychain, this suite cannot be namespaced away: it deletes the
real ~/Documents/mmkv and ~/Documents/ldk, and `cleanupRNKeychain` deletes RN
keychain items by service with no access-group filter. On a device mid-migration
from React-Native Bitkit, that is the migration source.

Skip it unless BITKIT_DESTRUCTIVE_TESTS=1, and set that only in the integration
lane, which already erases its simulator between attempts. Move the suite into
that lane's -only-testing list so it keeps running somewhere rather than being
silently dropped, and out of the unit lane.

Also drops -skip-testing:BitkitTests/TxBumpingTests, which names a class that no
longer exists.

The gate guards tearDown too: XCTest runs it even when setUp skips, and the
cleanup is the destructive part.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AcCTBgiMafXxw71MWBGB2T
The gate never received it. xcodebuild does not pass the invoking shell's
environment to the simulator-hosted test runner — only variables named
TEST_RUNNER_<VAR>, forwarded with the prefix stripped. So the guard read a
variable that never arrived, all twelve tests in the suite self-skipped in the
integration lane as well as the unit lane, and because skips do not fail a lane
both workflows stayed green with the suite running nowhere.

That is the outcome moving the suite into the integration lane was meant to
prevent. Verified both halves on a simulator:

  BITKIT_DESTRUCTIVE_TESTS=1 xcodebuild test …             1 test, 1 skipped
  TEST_RUNNER_BITKIT_DESTRUCTIVE_TESTS=1 xcodebuild test … 1 test, 0 skipped

The guard keeps reading the unprefixed name, which is what the runner sees. The
helper's documentation now says how to set it locally, since the obvious spelling
fails silently.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AcCTBgiMafXxw71MWBGB2T
…espace

`KeychainTests` only asserted that entries saved through `account(for:)` were gone
after a wipe, so it passed with or without the guard that keeps the wipe inside the
unit-test namespace. Removing the guard would wipe a simulator wallet again while
the suite stayed green.

Plant an account outside the namespace, wipe, and assert it survives. The probe
uses a unique throwaway name rather than a real key like `bip39_mnemonic_0`: on a
simulator with a wallet that account already exists, and deleting it by name in
teardown would remove the real seed.

Verified on an erased simulator: passes with the guard, fails with it removed
("wipeEntireKeychain deleted an account outside the unit-test namespace").

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AcCTBgiMafXxw71MWBGB2T
`testMarkerPathUsesSandboxDocuments` only checked that the marker sits under
Documents, which the real path already did, so removing the `unit-tests` redirect
would let `delete()` remove the real marker while every test passed.

Assert the path is inside `unit-tests`, and that the real `Documents/.bitkit_installed`
survives `create()` and `delete()`. The stand-in is planted only when no real
marker exists and only that stand-in is removed — on a simulator with a real
install, deleting that file makes the next launch wipe the keychain.

Verified on an erased simulator: both pass with the redirect and fail without it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AcCTBgiMafXxw71MWBGB2T
@jvsena42
jvsena42 force-pushed the fix/733-stop-test-suites-wiping-the-wallet branch from cc4ff06 to a8f95e5 Compare September 21, 2026 10:28
@jvsena42
jvsena42 merged commit b12e992 into master Sep 22, 2026
19 of 23 checks passed
@jvsena42
jvsena42 deleted the fix/733-stop-test-suites-wiping-the-wallet branch September 22, 2026 09:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: hardware transfer unit tests write mock transfers into the app's UserDefaults

3 participants