Fix set_and_wait_rc helper by clearing out stale events before new co…

[vlad-scherbich]

Motivation

set_and_wait_rc has a race condition: it clears the session after finding
telemetry/RC events, which means a subsequent call can pick up stale events from
the prior config. This causes flaky failures in multiple test_dynamic_configuration
tests (APMAPI-1051, APMAPI-734, APMAPI-1400, APMAPI-1860).

Follow-up to #6342, which
applied a per-test retry workaround. This PR fixes the root cause.

Changes

Clear the session before setting the new RC config in set_and_wait_rc, instead
of clearing after finding events. This ensures both wait_for_telemetry_event and
wait_for_rc_apply_state only observe events triggered by the new config.

Testing

• local run: the 6 errors are timeouts with local Docker containers, can be ignored

$ ./run.sh PARAMETRIC -L python tests/parametric/test_dynamic_configuration.py
...
11:39:18.944 ERROR    Timeout of 60 seconds exceeded waiting for HTTP server to start. Please check logs.
11:39:18.944 INFO     Stopping python-test-client-4f1b3e
--------------- generated xml file: /Users/vlad.scherbich/go/src/github.com/DataDog/system-tests/logs_parametric/reportJunit.xml ---------------
======================================== 6 passed, 20 skipped, 3 xfailed, 6 errors in 266.40s (0:04:26) ========================================

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

tests/parametric/test_dynamic_configuration.py                          @DataDog/system-tests-core @DataDog/apm-sdk-capabilities

[datadog-datadog-prod-us1-2]

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 11 Tests failed

tests.parametric.test_dynamic_configuration.TestDynamicConfigV1.test_apply_state[library_env0, parametric-php] from system_tests_suite (Datadog) (Fix with Cursor)

AssertionError: Telemetry event app-client-configuration-change not found

self = <tests.parametric.test_dynamic_configuration.TestDynamicConfigV1 object at 0x7f387e3ee660>
test_agent = <utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7f384b949c10>
test_library = <utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f384b427920>

    @parametrize("library_env", [{**DEFAULT_ENVVARS}])
    def test_apply_state(self, test_agent: TestAgentAPI, test_library: APMLibrary) -> None:
        """Create a default RC record and ensure the apply_state is correctly set.
    
...

tests.parametric.test_dynamic_configuration.TestDynamicConfigV1.test_apply_state[library_env0, parametric-ruby] from system_tests_suite (Datadog) (Fix with Cursor)

AssertionError: Telemetry event app-client-configuration-change not found

self = <tests.parametric.test_dynamic_configuration.TestDynamicConfigV1 object at 0x7f10d033ba70>
test_agent = <utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7f109b504bf0>
test_library = <utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f109b892000>

    @parametrize("library_env", [{**DEFAULT_ENVVARS}])
    def test_apply_state(self, test_agent: TestAgentAPI, test_library: APMLibrary) -> None:
        """Create a default RC record and ensure the apply_state is correctly set.
    
...

tests.parametric.test_dynamic_configuration.TestDynamicConfigV1.test_log_injection_enabled[library_env0, parametric-dotnet] from system_tests_suite (Datadog) (Fix with Cursor)

AssertionError: Telemetry event app-client-configuration-change not found

self = <tests.parametric.test_dynamic_configuration.TestDynamicConfigV1 object at 0x7f43133220f0>
test_agent = <utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7f42e0742810>
test_library = <utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f42e07437a0>

    @missing_feature(
        context.library in ("cpp", "golang"),
        reason="Tracer doesn't support automatic logs injection",
    )
...

View all

ℹ️ Info

❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 8c987a4 | Docs | Datadog PR Page | Was this helpful? Give us feedback!}

[KowalskiThomas]

No context, but approving to unblock.

[vlad-scherbich]

Closing in favor of #6371