fix(database): recover from Room's silent connection-pool wedge (#6608) - #6658
Conversation
📝 WalkthroughWalkthroughThe change adds database-flow retry handling, stalled-flow detection, sliding-window recovery limits, detached-pool cleanup, and regression tests. Data-source flows now use the retry operator for recoverable Room pool failures. ChangesDatabase flow recovery
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant DataSource
participant DatabaseManager
participant DatabasePool
DataSource->>DatabaseManager: observe current database flow
DatabaseManager->>DatabasePool: acquire pool and collect query
DatabasePool-->>DatabaseManager: emit data or pool failure
DatabaseManager->>DatabasePool: relatch replacement pool after recovery
DatabaseManager-->>DataSource: publish recovered flow value
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 6✅ Passed checks (6 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This comment has been minimized.
This comment has been minimized.
❌ 11 Tests Failed:
View the top 2 failed test(s) by shortest run time
View the full list of 9 ❄️ flaky test(s)
To view more test analytics, go to the Test Analytics Dashboard |
|
@coderabbitai full review |
|
|
@coderabbitai full review |
✅ Action performedFull review finished. |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
core/database/src/commonTest/kotlin/org/meshtastic/core/database/DbFlowRecoveryTest.kt (1)
38-104: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse Turbine for the changed Flow tests.
core/database/src/commonTest/kotlin/org/meshtastic/core/database/DbFlowRecoveryTest.kt#L38-L104: replacetoList()-based Flow assertions with Turbine assertions.core/database/src/commonTest/kotlin/org/meshtastic/core/database/DatabaseManagerShutdownTest.kt#L372-L505: use Turbine to collect and assert Flow events.core/data/src/commonTest/kotlin/org/meshtastic/core/data/datasource/PoisonedPoolNodeFlowRecoveryTest.kt#L54-L84: use Turbine for recovery and eager-sharing Flow assertions.As per coding guidelines, “Use Turbine for Flow testing.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@core/database/src/commonTest/kotlin/org/meshtastic/core/database/DbFlowRecoveryTest.kt` around lines 38 - 104, Replace toList()-based Flow assertions with Turbine event assertions in DbFlowRecoveryTest.kt lines 38-104, DatabaseManagerShutdownTest.kt lines 372-505, and PoisonedPoolNodeFlowRecoveryTest.kt lines 54-84. Use Turbine to await expected emissions, completion, and failures while preserving each test’s existing recovery, shutdown, and eager-sharing expectations.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@core/database/src/commonMain/kotlin/org/meshtastic/core/database/DatabaseManager.kt`:
- Around line 1035-1051: Update trimDetachedDatabasesLocked and its
detached-pool lifecycle tracking so pools are not closed while DAO Flow
collectors or holders of earlier currentDb values may still be active. Track
each detached Flow generation through cancellation completion, and only make a
pool trimmable after all associated Flow users finish; otherwise retain detached
pools until close(). Do not use replacement count or
hasActiveDatabaseAccessLocked alone as proof of safety.
---
Nitpick comments:
In
`@core/database/src/commonTest/kotlin/org/meshtastic/core/database/DbFlowRecoveryTest.kt`:
- Around line 38-104: Replace toList()-based Flow assertions with Turbine event
assertions in DbFlowRecoveryTest.kt lines 38-104, DatabaseManagerShutdownTest.kt
lines 372-505, and PoisonedPoolNodeFlowRecoveryTest.kt lines 54-84. Use Turbine
to await expected emissions, completion, and failures while preserving each
test’s existing recovery, shutdown, and eager-sharing expectations.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: ac983c02-b08d-4a50-a8e9-b9275d90bb32
📒 Files selected for processing (12)
core/data/src/commonMain/kotlin/org/meshtastic/core/data/datasource/DeviceLinkLocalDataSource.ktcore/data/src/commonMain/kotlin/org/meshtastic/core/data/datasource/EventFirmwareEditionLocalDataSource.ktcore/data/src/commonMain/kotlin/org/meshtastic/core/data/datasource/SwitchingChannelSetDataSource.ktcore/data/src/commonMain/kotlin/org/meshtastic/core/data/datasource/SwitchingNodeInfoReadDataSource.ktcore/data/src/commonMain/kotlin/org/meshtastic/core/data/repository/NodeRepositoryImpl.ktcore/data/src/commonTest/kotlin/org/meshtastic/core/data/datasource/PoisonedPoolNodeFlowRecoveryTest.ktcore/data/src/commonTest/kotlin/org/meshtastic/core/data/repository/RadioConfigRepositoryImplTest.ktcore/database/src/commonMain/kotlin/org/meshtastic/core/database/DatabaseManager.ktcore/database/src/commonMain/kotlin/org/meshtastic/core/database/DbFlowRecovery.ktcore/database/src/commonTest/kotlin/org/meshtastic/core/database/DatabaseManagerShutdownTest.ktcore/database/src/commonTest/kotlin/org/meshtastic/core/database/DatabaseManagerTestFixture.ktcore/database/src/commonTest/kotlin/org/meshtastic/core/database/DbFlowRecoveryTest.kt
|
Addressed the Turbine nitpick in addc45d.
One wrinkle worth recording for anyone writing Flow tests against retry/backoff code here: Turbine's await deadline runs on the test scheduler's virtual clock, which the retry backoff also advances, so the default 1s deadline trips as soon as a retry sleeps. Both suites pass an explicit Left 🤖 Addressed by Claude Code |
addc45d to
f7fa095
Compare
This comment has been minimized.
This comment has been minimized.
shard-core failure is pre-existing on
|
| commit | :core:ui:jvmTest + :feature:connections:jvmTest |
|---|---|
28dd35ef9 (before the bump) |
BUILD SUCCESSFUL |
bc4e9da3a (#6662, CMP 1.12.0-rc01) |
fails |
d5848ad5e (current main) |
fails |
Two things explain why this went unnoticed: #6662 merged with all three test shards cancelled, so these tests never gated it; and the shard that ran on #6661 afterwards replayed :core:service, :feature:connections, and :feature:node test tasks FROM-CACHE, so it never executed them either. Current main's green tick is therefore not evidence that main is healthy.
Because the cache has been masking this, the true blast radius is unknown — any task not re-executed since bc4e9da3a could also be affected. Establishing that needs a --no-build-cache --rerun-tasks run at d5848ad5e across all shards.
Not enqueueing this PR until main is green, since the queue would eject it for a failure it didn't cause. No fix pushed here on purpose: the regression belongs to #6662 and should be reverted or fixed forward on its own.
🤖 Addressed by Claude Code
This comment has been minimized.
This comment has been minimized.
Correction: I was wrong above — at least part of this is mineMy previous comment attributed the shard-core failure to #6662. That is not supported, and I'm retracting it. I checked CI runs on
So that test passes on main and fails on this branch. My earlier evidence came from a local run whose environment disagrees with CI on shard-feature on this PR also fails Working hypothesis, not yet confirmed: Investigating now. This PR should not be merged in its current state. 🤖 Addressed by Claude Code |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Final attribution, with the experiments that settle itMy retraction above said "at least part of this is mine." Completing the isolation shows that was also wrong — but for a subtler reason than my first comment. Both of my earlier attributions failed the same way: they trusted whole-module runs of order-dependent tests. Run filtered (single class,
Both classes fail deterministically on clean main when they run without the rest of their module in front of them. In whole-module runs the outcome flips with test ordering — which is why CI showed them green at Nothing in So the overall state of main has two independent problems, neither from this PR:
57943ed (narrowing the retry predicate to the actual wedge signature) stays: it's correct regardless — a closed pool is handled by re-latching, and retrying it kept flows alive past their scope. The modules this PR touches remain green: 738 jvm tests, 0 failures, forced. Holding enqueue until main is actually green under real execution. 🤖 Addressed by Claude Code |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Room 3.0.1 logs its pool acquire-timeout and retries forever instead of throwing, so a leaked permit makes DAO flows hang with no exception and no value. Every catch-based recovery path was therefore unreachable, and the node list stayed empty for the process lifetime. Detect the wedge by absence: bound the first emission of each re-latched DAO flow and route a stall into the existing pool-reopen path. Restart eagerly-shared repository flows after a recoverable pool failure so a terminal upstream can no longer kill a process-lifetime StateFlow. Replace both manager-lifetime recovery caps with one sliding rate window, per origin, inside the seam #6661 introduced. A lifetime budget is spent early when a wedge recurs within minutes, after which every later wedge is permanent -- the failure #6608 reports. The wedged-pool quarantine expires with the same window, without which the budget and the admission gate block each other and the pool stays write-dead for the process. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Retrying a closed pool kept a flow alive that used to terminate, leaving a backoff loop inside a scope that had already gone away — it surfaced as a coroutine resuming on a reset Main dispatcher in unrelated ViewModel tests. observeCurrentDb already re-latches onto the replacement pool, so retrying a closed pool adds nothing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2dac3fa to
b029602
Compare
Why
Field reports in #6608 describe a permanent wedge on 2.8.1-SNAPSHOT: the node list empties and never reloads, updates stop, and only a force-stop recovers. The attached logcats pinned it to a leaked permit in Room 3.0.1's
ConnectionPoolImpl— but reading them closely changes the diagnosis in a way that matters:That is
W/System.err— aprintStackTrace()from Room'sLOG_TIMEOUT_EXCEPTIONmode, not an exception delivered to us. Across the three logs there are 156 dumps in a single second and zero app-side recovery log lines. Room'sacquireWithTimeoutlogs the timeout and loops forever, so a wedged pool produces neither a value nor a failure: every DAO flow simply hangs.Everything we had built for this failure keyed off a thrown
SQLException, so none of it could ever fire — the recovery caps inDatabaseManagerwere a red herring, not the cause. On top of that,NodeRepositoryImpl.nodeDBbyNumisstateIn(processLifecycle, SharingStarted.Eagerly, …); once its upstream terminates, the sharing coroutine is dead for the process and re-navigation cannot restart it. That is the "node list disappears and never reloads" symptom.setSingleConnectionPool()and thelimitedParallelism(1)query context are unchanged — both remain load-bearing (see theconfigureCommonKDoc).🐛 Fixes
observeCurrentDbnow bounds the first emission of each re-latched DAO flow (45s, comfortably above any legitimate cold-open/migration/backfill-contended query, and above Room's own 30s acquire timeout). A missing first emission raisesDbFlowStalledException, which routes into the existing pool-reopen recovery. Only the first emission per latch is bounded — later emissions are event-driven and legitimately sparse.MAX_FLOW_POOL_RECOVERIES_PER_MANAGER_LIFETIMEbecameMAX_FLOW_POOL_RECOVERIES_PER_WINDOWover a sliding 10-minute window. Reporters see wedges "multiple times within minutes", which exhausted the old lifetime budget and made every later wedge permanent by construction.retryOnDbPoolFailurerestarts a database-backed flow after a recoverable pool failure with capped exponential backoff (reset on a successful emission; non-database failures still propagate). Applied to everyobserveCurrentDbcall site, so a terminal failure can no longer kill a process-lifetimeSharingStarted.EagerlyStateFlow.🧹 Cleanups
RadioConfigRepositoryImplTestbuildsSwitchingChannelSetDataSourcewith the realFakeDatabaseProviderinstead of a mokkery autofill mock, which returnsnullfor the non-null DAO flow.Known remaining gap (not fixed here)
The one-shot write path has the same exposure and is not addressed:
withDbruns its callback underNonCancellableondispatchers.io.limitedParallelism(1), so a callback wedged inside Room can neither be cancelled nor timed out, and every laterwithDbqueues behind it forever. Reopening the pool does not help because the lane never drains. That is the likely cause of the "disconnect impossible" half of #6608 and needs its own design — a bounded-wait scheme or a per-pool lane — rather than a timeout that cannot interrupt aNonCancellableblock.The TAK-server/session-lease backpressure wedge that drives the cancellation churn is being handled on a separate track.
Testing Performed
Baseline gate green:
spotlessApply spotlessCheck detekt assembleDebug test allTests kmpSmokeCompile.New tests:
DatabaseManagerShutdownTest.silentlyStalledFlowIsRecoveredWithoutAnyThrownFailure— a DAO flow that never emits and never fails (exactly what a leaked permit looks like) is recovered.…quietFlowAfterAFirstEmissionIsNotTreatedAsStalled— a flow that emits once then stays quiet for 3× the deadline does not trigger recovery.…flowPoolRecoveryResumesAfterTheRateWindowClears— the [Bug]: Stale node connection still occurring after fix for #6491 #6608 regression: after the budget is spent, a later wedge still recovers once the window clears.…recurringFlowPoolRecoveryNeverClosesDetachedPoolsBeforeShutdown— recovery retains every replaced pool and onlyclose()reclaims them.DbFlowRecoveryTest— restart on acquire-timeout and closed-pool failures, propagation of unrelated failures, backoff reset after a successful emission.PoisonedPoolNodeFlowRecoveryTest— poisons the pool for far more failures than any recovery budget allows and provesnodeDBbyNumFlow,myNodeInfoFlow, and an eagerly-shared StateFlow over them still reach a value.Rebased onto #6661 — and this changes one policy it just shipped
#6661 landed first and rewrote
withDb, so this is rebased onto it. ItshasReachedRecoveryLimit(origin)/recordRecovery(origin)seam is kept and my sliding window moved inside it, which is cleaner than either branch alone.The policy change, stated plainly: #6661 bounded wedge recovery with
MAX_WEDGE_POOL_RECOVERIES_PER_MANAGER_LIFETIME = 3and quarantined the pool once spent. That is the same lifetime-budget shape this PR removes for the Flow path, and it has the same consequence. The quarantine clears in only three places —closeInactiveDatabase, the reopen-replacement path, andclose(). The quarantined pool is_currentDb, so it is never an inactive eviction candidate, and the reopen path is exactly what a spent budget refuses. So after three wedges in one process, writes fail fast until a device switch or a force-stop. Given #6608 reporters see wedges recurring within minutes, three arrives quickly.Both budgets are now one sliding window (
POOL_RECOVERY_WINDOW_MS), with per-origin deques and per-origin limits (6 Flow, 3 wedge). Per-origin so a Flow-recovery storm cannot starve the write path's budget; limits kept distinct so #6661's chosen sensitivity is preserved rather than silently re-tuned.Quarantine expiry is the load-bearing half. Windowing the budget alone would change nothing observable: recovery is refused until a write is admitted, and writes are refused until recovery happens. The quarantine therefore expires with the same window. It is evaluated in the quarantine map rather than against the budget deques because the gate holds
writerTrackerMutexwhile the deques are guarded bymutex, and the established order ismutex→writerTrackerMutex; reading the budget there would invert it.Expiry re-admits against the still-wedged pool, since nothing replaced it — so the first write back pays one
WITH_DB_TIMEOUT_MSdeadline and its abandonment publishes the replacement. That sacrificial write is the guaranteed floor; when DAO Flows are collecting, a stall recovery usually replaces the pool sooner.writesRecoverAfterTheWedgeWindowClearspins the whole cycle rather than stopping at re-admission, which would pass even if recovery never completed.History: the CMP 1.12.0-rc01 detour
Earlier revisions of this description reported main as red and attributed CI failures across three theories. Final resolution: the failures were regressions from the CMP 1.12.0-rc01 bump (#6662), which merged with its test shards cancelled and was masked afterwards by build-cache replay. It was reverted in #6664, and this branch is rebased onto that revert (
d8361ccd1). The full attribution trail — including two incorrect attributions by this PR's author, and their corrections — is preserved in the comments.The one known residual flake repo-wide is
NodeDetailCompassLifecycleTest(~66% per run, CMP-independent,waitUntiltimeout); if this PR merges only after a retry that flipped it, the merge report will say so.Review follow-up
CodeRabbit caught a real defect in the first push: I had added trimming of detached (replaced) pools to bound memory, gated on the reader/writer access counters. Those counters only cover
withReadDb/withDb. DAO Flow collectors are untracked, and Paging factories latch a rawcurrentDb.value—PacketRepositoryImpl.kt:69builds its paging source that way — so trimming could close a pool a livePagingSourcestill held and surfaceConnection pool is closed. That would have traded a recoverable wedge for a new crash.Trimming is removed; the documented invariant (only
close()reclaims detached pools) is restored, and the recovery rate window is what bounds churn. The trade-off is explicit: under a repeated wedge, replaced pools accumulate until shutdown, capped in rate rather than in total. Correctness over memory — an idle detached pool costs one SQLite connection and a statement cache, while closing one out from under a Paging source breaks the message list.One pre-existing test (
flowPoolRecoveryStopsAfterIntermittentFailuresFillTheRateWindow) caught a real defect in the first draft of this change: the recovery counter reset sat downstream of the new channel hop, where an upstream failure can overtake the value emitted a moment earlier. The reset now runs in the producing coroutine.Not validated on a device — the wedge is a timing-dependent field failure we have not reproduced locally.
Upstream draft — Google Issue Tracker filing for the Room bug (for James to file; not submitted)
Google Issue Tracker draft — DO NOT FILE AUTOMATICALLY
Component: Android Public Tracker > Android Public Tracker > Jetpack (androidx) > Room
Type: Bug · Version: androidx.room3 3.0.1 (also present in 3.0.0)
Title: ConnectionPoolImpl leaks a connection permit when an acquire times out after
acquire()alreadysucceeded, permanently wedging the pool (
permits=0with all connectionsFree)Summary
Pool.acquireWithTimeoutcan drop a successfully acquiredConnectionWrapperwithout recycling it, losing thesemaphore permit that the acquisition consumed. With
setSingleConnectionPool()(capacity = 1) the pool becomespermanently unusable: every subsequent
useConnectionwaits the full 30s timeout, and because the pool's defaultonTimeoutmode isLOG_TIMEOUT_EXCEPTION,acquireWithTimeout'swhile (true)loop retries forever. Theapplication sees neither a value nor an exception — every database read hangs for the life of the process, while
logcat fills with the acquire-timeout dump.
We are hitting this in production in Meshtastic-Android (KMP,
BundledSQLiteDriver, Android + JVM desktop). Fieldreports describe a permanent app wedge that only a force-stop clears.
Analysis
ConnectionPoolImpl.kt(3.0.1),Pool.acquireWithTimeout:The defect is the interaction of (1), (2) and (3):
acquire()consumes a permit fromconnectionPermitsand returns a wrapper. The assignment to the outerconnectionvar therefore happens beforewithTimeoutreturns.withTimeoutstill throwsTimeoutCancellationExceptionif its deadline elapses as the block completes. Theresult is
exceptionThrown is TimeoutCancellationExceptionandconnection != null.onTimeoutisConnectionPoolImpl.onTimeout, whose behavior depends on theinternal var onTimeoutmode. The default isLOG_TIMEOUT_EXCEPTION, which callsex.printStackTrace()and returns normally.catchat (3) — the only place a strandedconnectionis recycled — never runs.The loop iterates,
connectionis re-initialized tonull, and the acquired wrapper is unreachable. Its permitis never released.
Only
THROW_TIMEOUT_EXCEPTIONmode is safe here, and that mode is not publicly configurable(see the
TODO(b/404380974)aboveinternal var timeout).Two consequences:
capacity = 1(single-connection pool) means one leak wedges the pool forever. A multi-readerpool degrades one permit at a time toward the same end state.
LOG_TIMEOUT_EXCEPTIONmodeacquireWithTimeoutneverreturns and never throws, so
useConnection— and every DAO query/Flow above it — hangs indefinitely. An appcannot detect or recover from this through normal error handling; it can only infer it from a missing result.
Cancellation churn makes the race reachable in practice: our DB flows are re-latched via
flatMapLateston deviceswitches, so acquisitions are routinely cancelled mid-flight (the dumps below show the requesting coroutine in
Cancellingstate).Evidence (production logcat, Meshtastic-Android 2.8.1)
The dump is self-consistent with the analysis and, as far as we can tell, with nothing else:
permits=0— the single permit is gone.queue=(size=0)[]— the connection is not inavailableConnections, sorecycle()was never called for it.Status: Free connection—ConnectionWrapper.dumpprints this only whenacquireCoroutineContext == null.markAcquired(...)is applied inuseConnectionafteracquireWithTimeoutreturns, so a wrapper that wasacquired inside
acquire()but dropped by the timeout branch is exactly one that showsFreewhile its permit isheld. A connection genuinely in use would print
Coroutine: [...]instead.W/System.errwith the frameConnectionPoolImpl.onTimeoutreached fromPool.acquireWithTimeoutconfirmsLOG_TIMEOUT_EXCEPTION(printStackTrace), i.e. the exception is never delivered to the caller.waiter in the app is parked on the wedged pool.
Once wedged, the state never clears — reported repeatedly, and only force-stopping the app restores function.
Suggested fix
In
Pool.acquireWithTimeout, recycle an acquired connection whenever the iteration is not going to return it,independently of whether
onTimeoutthrows:Equivalently: move the recycle into a
finallythat runs unless the wrapper is being returned.Secondary requests:
onTimeout/timeoutpublicly configurable (b/404380974).LOG_TIMEOUT_EXCEPTIONturns a pool-levelfault into an unbounded silent hang, which application code cannot recover from. At minimum, consider making
THROW_TIMEOUT_EXCEPTIONthe default, or bounding thewhile (true)retry loop.permits == capacitywhen every connection isFree, so a leaked permit issurfaced as a diagnosable error rather than an infinite wait.
Environment
BundledSQLiteDriversetSingleConnectionPool()andsetQueryCoroutineContext(ioDispatcher.limitedParallelism(1))Flows re-latched withflatMapLateston database switches (the cancellation churn)Notes for James before filing
30935733). 30935732 downloaded as a 9-byte file for me — re-fetch it from the issue before attaching.
If the Room team asks for a repro, the shape to try is:
capacity = 1pool, a coroutine that cancels acquisitionsin a tight loop, and
timeoutreduced so thewithTimeout-completes-as-deadline-elapses window is hit often.