[RC] Fix experiment state races during database loading - #16487
[RC] Fix experiment state races during database loading#16487ryannair05 wants to merge 1 commit into
Conversation
Protect experiment snapshots with generation-based synchronization so delayed database loads cannot overwrite newer state. Replace experiment rows transactionally and wait for queued persistence before updating A/B Testing.
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. |
Summary
aggregate generations
active experiment state
persistence before updating A/B Testing
Fixes #16303.
Root cause
The initial experiment database load can race with fetch and activation.
Synchronizing collection access prevents concurrent mutation, but does not by
itself ensure that fetched payloads, metadata, and active payloads belong to one
coherent logical state.
A single database-load flag also cannot distinguish between the three
independently updated fields. It can therefore either publish stale state or
discard a valid database value for an unrelated field. Activation can likewise
begin with one payload/metadata snapshot and commit after either part has
changed.
Implementation
The experiment state is stored as immutable snapshots protected by
os_unfair_lock. No JSON parsing, database access, or A/B Testing work occurswhile the lock is held.
Each independently updated field has its own generation:
The database load records those generations when it begins and only publishes a
field if that field has not changed in the meantime. This preserves valid
database results while rejecting only stale fields.
An aggregate state generation keeps activation coherent across fields.
Activation calculates the latest experiment start time from one immutable
payload/metadata snapshot and retries if the state changes before that snapshot
can be committed.
Experiment rows for a key are replaced inside a SQLite transaction, preventing
a failed multi-row replacement from leaving partial results. A/B Testing is
invoked only after the queued metadata write and active-payload replacement have
finished, so it cannot observe activation while persistence is still pending.
Relationship to #16326
The implementation in #16326 improves collection safety, but its single
database-load flag does not distinguish independently updated payloads,
metadata, and active payloads. It also does not make the payload and metadata
used by activation one validated logical snapshot.
The per-field generations cover the first case without discarding unrelated
database state. The aggregate generation and retry cover the second case by
preventing activation from committing a mixture of old and new state. The
transaction and completion ordering extend those consistency guarantees through
experiment persistence and the A/B Testing update boundary.
Regression tests
Five deterministic tests exercise the relevant interleavings:
metadata.
calculation.
experiments.
write have completed.
Each regression failed with its corresponding fix disabled and passed after the
fix was restored. The rollback and persistence-ordering tests were also run with
the database-initialization prerequisite enabled in isolation, ensuring their
failures reached the intended assertions instead of being masked by an earlier
failure.
Validation
race report.
RemoteConfigUniton iOS, macOS, Mac Catalyst, watchOS, and visionOS:132 tests and 0 failures on each platform. The live-console integration test
was excluded because it requires Firebase credentials.
scripts/check.sh --test-only HEAD: passed.API changes
None. The added database operation is internal to the Remote Config target.