Skip to content

Comments

Attempt to fix race conditions in 'set_and_wait_rc' by counting telem…#6371

Draft
vlad-scherbich wants to merge 1 commit intomainfrom
vlad/fix-dynamic-config-flakes
Draft

Attempt to fix race conditions in 'set_and_wait_rc' by counting telem…#6371
vlad-scherbich wants to merge 1 commit intomainfrom
vlad/fix-dynamic-config-flakes

Conversation

@vlad-scherbich
Copy link
Contributor

@vlad-scherbich vlad-scherbich commented Feb 23, 2026

Motivation

~25 of 32 system test failures on dd-trace-py main in 2026 stem from dynamic configuration tests. Root cause: a race in set_and_wait_rc—it can match stale app-client-configuration-change telemetry events from a previous RC update and return before the new config is actually applied.

Changes

  • set_and_wait_rc: Replace wait_for_telemetry_event(..., clear=True) with telemetry event counting. Snapshot the count of app-client-configuration-change events before setting RC, then poll until the count increases—guaranteeing we match a genuinely new event, not a stale one.
  • test_capability_tracing_sample_rules: Increase wait_loops from 100 to 400 (~4s) so the library has enough time to send its first RC request.
  • _MAX_RC_EVENT_WAIT_LOOPS: Add module-level constant (400) for the RC event polling loop instead of a magic number.

Files modified: tests/parametric/test_dynamic_configuration.py

Workflow

  1. ⚠️ Create your PR as draft ⚠️
  2. Work on your PR until the CI passes
  3. Mark it as ready for review
    • Test logic is modified? -> Get a review from RFC owner.
    • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

  • Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
  • A docker base image is modified?
    • the relevant build-XXX-image label is present
  • A scenario is added, removed or renamed?

@github-actions
Copy link
Contributor

github-actions bot commented Feb 23, 2026

CODEOWNERS have been resolved as:

utils/scripts/list_flaky_system_tests.sh                                @DataDog/system-tests-core
tests/parametric/test_dynamic_configuration.py                          @DataDog/system-tests-core @DataDog/apm-sdk-capabilities
utils/docker_fixtures/_test_agent.py                                    @DataDog/system-tests-core

@datadog-official

This comment has been minimized.

@vlad-scherbich vlad-scherbich force-pushed the vlad/fix-dynamic-config-flakes branch from 8842b4c to 1efc4da Compare February 23, 2026 20:52
@vlad-scherbich vlad-scherbich force-pushed the vlad/fix-dynamic-config-flakes branch from 1efc4da to a8cf21a Compare February 23, 2026 21:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant