Skip to content

[RC] Fix experiment state races during database loading - #16487

Open
ryannair05 wants to merge 1 commit into
firebase:mainfrom
ryannair05:agent/remote-config-experiment-race
Open

[RC] Fix experiment state races during database loading#16487
ryannair05 wants to merge 1 commit into
firebase:mainfrom
ryannair05:agent/remote-config-experiment-race

Conversation

@ryannair05

@ryannair05 ryannair05 commented Aug 8, 2026

Copy link
Copy Markdown

Summary

  • synchronize immutable Remote Config experiment snapshots with per-field and
    aggregate generations
  • prevent delayed database loads from overwriting newer fetched, metadata, or
    active experiment state
  • replace experiment payload rows transactionally and wait for queued
    persistence before updating A/B Testing

Fixes #16303.

Root cause

The initial experiment database load can race with fetch and activation.
Synchronizing collection access prevents concurrent mutation, but does not by
itself ensure that fetched payloads, metadata, and active payloads belong to one
coherent logical state.

A single database-load flag also cannot distinguish between the three
independently updated fields. It can therefore either publish stale state or
discard a valid database value for an unrelated field. Activation can likewise
begin with one payload/metadata snapshot and commit after either part has
changed.

Implementation

The experiment state is stored as immutable snapshots protected by
os_unfair_lock. No JSON parsing, database access, or A/B Testing work occurs
while the lock is held.

Each independently updated field has its own generation:

  • fetched experiment payloads
  • experiment metadata
  • active experiment payloads

The database load records those generations when it begins and only publishes a
field if that field has not changed in the meantime. This preserves valid
database results while rejecting only stale fields.

An aggregate state generation keeps activation coherent across fields.
Activation calculates the latest experiment start time from one immutable
payload/metadata snapshot and retries if the state changes before that snapshot
can be committed.

Experiment rows for a key are replaced inside a SQLite transaction, preventing
a failed multi-row replacement from leaving partial results. A/B Testing is
invoked only after the queued metadata write and active-payload replacement have
finished, so it cannot observe activation while persistence is still pending.

Relationship to #16326

The implementation in #16326 improves collection safety, but its single
database-load flag does not distinguish independently updated payloads,
metadata, and active payloads. It also does not make the payload and metadata
used by activation one validated logical snapshot.

The per-field generations cover the first case without discarding unrelated
database state. The aggregate generation and retry cover the second case by
preventing activation from committing a mixture of old and new state. The
transaction and completion ordering extend those consistency guarantees through
experiment persistence and the A/B Testing update boundary.

Regression tests

Five deterministic tests exercise the relevant interleavings:

  1. A delayed database load cannot overwrite a newer fetched payload.
  2. A delayed database load cannot overwrite newer activated payloads or
    metadata.
  3. Activation retries if the database load publishes metadata during its
    calculation.
  4. A failed multi-row replacement rolls back without leaving partially replaced
    experiments.
  5. A/B Testing is not updated until the active payload and preceding metadata
    write have completed.

Each regression failed with its corresponding fix disabled and passed after the
fix was restored. The rollback and persistence-ordering tests were also run with
the database-initialization prerequisite enabled in isolation, ensuring their
failures reached the intended assertions instead of being masked by an earlier
failure.

Validation

  • Focused regression suite with Thread Sanitizer: 5 tests, 0 failures, and no
    race report.
  • RemoteConfigUnit on iOS, macOS, Mac Catalyst, watchOS, and visionOS:
    132 tests and 0 failures on each platform. The live-console integration test
    was excluded because it requires Firebase credentials.
  • CocoaPods 1.17 iOS lint with analysis and tests: passed.
  • scripts/check.sh --test-only HEAD: passed.
  • Repository pre-commit style checks with clang-format 22: passed.

API changes

None. The added database operation is internal to the Remote Config target.

Protect experiment snapshots with generation-based synchronization so delayed database loads cannot overwrite newer state. Replace experiment rows transactionally and wait for queued persistence before updating A/B Testing.
@gemini-code-assist

Copy link
Copy Markdown
Contributor
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

@ryannair05 ryannair05 closed this Aug 8, 2026
@ryannair05
ryannair05 deleted the agent/remote-config-experiment-race branch August 8, 2026 03:20
@ryannair05
ryannair05 restored the agent/remote-config-experiment-race branch August 8, 2026 03:25
@ryannair05 ryannair05 reopened this Aug 8, 2026
@ryannair05
ryannair05 marked this pull request as ready for review August 8, 2026 03:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RemoteConfig: activate() crashes in RCNConfigExperiment due to a data race with the experiment DB load

1 participant