test(debugger): ensure span error budgets are reasonable

[tylfin]

Motivation

Add a test to verify that span decoration probes with improper expressions (e.g., "Cannot dereference field") have their error logs rate-limited to at most once per second. Currently marked as a bug for Java since span decoration probes lack probe budgets, producing 150 error snapshots for 150 requests instead of the expected ~1 per second.

Changes

• Added a new test method test_span_probe_expression_budgets in tests/debugger/test_debugger_probe_budgets.py to verify that error logs from span probe expression failures are rate-limited to at most 1 per second, with appropriate assertions and handling for language-specific limitations.
• Introduced a helper method setup_span_probe_expression_budgets to configure and trigger the span probe expression budget scenario in the same test file.
• Added a new probe configuration file probe_span_method_budgets_expression.json under tests/debugger/utils/probes/ to define a span probe with an intentionally failing expression, used for testing error logging and rate limiting.
• Imported the bug decorator from utils in tests/debugger/test_debugger_probe_budgets.py to allow marking tests with known bugs, specifically for Java.

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

tests/debugger/utils/probes/probe_span_method_budgets_expression.json   @DataDog/debugger @DataDog/system-tests-core
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
tests/debugger/test_debugger_probe_budgets.py                           @DataDog/debugger @DataDog/system-tests-core

[nccatoni]

LGTM for @DataDog/system-tests-core, you should also get a review from someone familiar with the feature

[datadog-official]

⚠️ Tests

✨ Fix all issues with Cursor

⚠️ Warnings

❄️ 2 New flaky tests detected

tests.debugger.test_debugger_capture_expressions.Test_Debugger_Method_Capture_Expressions.test_log_method_capture_expressions[spring-boot-wildfly] from system_tests_suite (Datadog) (Fix with Cursor)

AssertionError: assert 'Snapshot was not received' is None
 +  where 'Snapshot was not received' = <built-in method join of str object at 0x7f6170c2aa60>(['Snapshot was not received'])
 +    where <built-in method join of str object at 0x7f6170c2aa60> = '\n'.join
 +    and   ['Snapshot was not received'] = <tests.debugger.test_debugger_capture_expressions.Test_Debugger_Method_Capture_Expressions object at 0x7f616293ddc0>.setup_failures

self = <tests.debugger.test_debugger_capture_expressions.Test_Debugger_Method_Capture_Expressions object at 0x7f616293ddc0>

    def test_log_method_capture_expressions(self):
>       self._assert()

...

tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots.test_span_method_snapshot[spring-boot-wildfly] from system_tests_suite (Datadog) (Fix with Cursor)

ValueError: Trace span7359-08d8-4cee-937f-329acb4c20ff was not received.

self = <tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots object at 0x7fcf403ad910>

    @missing_feature(context.library == "golang", reason="Not yet implemented", force_skip=True)
    def test_span_method_snapshot(self):
        self._assert()
>       self._validate_spans()

tests/debugger/test_debugger_probe_snapshot.py:112: 
...

🧪 3 Tests failed

❄️ Known flaky: tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Line_Probe_Snaphots.test_span_decoration_line_snapshot[spring-boot-jetty] from system_tests_suite (Datadog) (Fix with Cursor)

ValueError: Trace decord62-1aab-462f-a595-0ecc9e5337b7 was not received.

self = <tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Line_Probe_Snaphots object at 0x7f5b452cc740>

    @missing_feature(context.library == "nodejs", reason="Not yet implemented", force_skip=True)
    @missing_feature(context.library == "golang", reason="Not yet implemented", force_skip=True)
    def test_span_decoration_line_snapshot(self):
        self._assert()
>       self._validate_spans()

...

❄️ Known flaky: tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots.test_span_decoration_method_snapshot[spring-boot-jetty] from system_tests_suite (Datadog) (Fix with Cursor)

ValueError: Trace decor59f-4c70-48cb-aa1a-00fea8f13e59 was not received.

self = <tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots object at 0x7f5b4529d5e0>

    @missing_feature(context.library == "golang", reason="Not yet implemented", force_skip=True)
    def test_span_decoration_method_snapshot(self):
        self._assert()
>       self._validate_spans()

tests/debugger/test_debugger_probe_snapshot.py:126: 
...

❄️ Known flaky: tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots.test_span_decoration_method_snapshot[spring-boot-payara] from system_tests_suite (Datadog) (Fix with Cursor)

ValueError: Trace decor650-2961-4797-b164-e52a5cb181c1 was not received.

self = <tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots object at 0x7f35d1c3eed0>

    @missing_feature(context.library == "golang", reason="Not yet implemented", force_skip=True)
    def test_span_decoration_method_snapshot(self):
        self._assert()
>       self._validate_spans()

tests/debugger/test_debugger_probe_snapshot.py:126: 
...

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 949efe2 | Docs | Datadog PR Page | Was this helpful? Give us feedback!}