ci: upload system-tests results to Test Optimization and add tracer-release nightly

[nccatoni]

Description

Two changes to the system-tests CI integration:

1. Upload all system-tests results to Test Optimization — adds an after_script to the .system_tests base job that copies JUnit XML files and uploads them via the existing silent-upload-junit-to-datadog.sh script.

2. Add tracer-release nightly — create a new system-tests workflow that uses the official system-tests reusable workflow to run the tracer-release scenario group on master.

This is meant to replace the system-tests-dashboard nightly.

Reviewer checklist

• This is a CI-only change, no code changes.
• Appropriate labels assigned.

[datadog-official]

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 1028 Tests failed

testSearchPhpBinaries from integration.DDTrace\Tests\Integration\PHPInstallerTest (Datadog) (Fix with Cursor)

DDTrace\Tests\Integration\PHPInstallerTest::testSearchPhpBinaries
Test code or tested code printed unexpected output: Searching for available php binaries, this operation might take a while.

testSimplePushAndProcess from laravel-58-test.DDTrace\Tests\Integrations\Laravel\V5_8\QueueTest (Datadog) (Fix with Cursor)

DDTrace\Tests\Integrations\Laravel\V5_8\QueueTest::testSimplePushAndProcess
Test code or tested code printed unexpected output: spanLinksTraceId: 69a969a600000000c48b65f113553381
tid: 69a969a600000000
hexProcessTraceId: c48b65f113553381
hexProcessSpanId: 2b288e9916edfc1a
processTraceId: 14162525539282531201
processSpanId: 3109892330865163290

phpvfscomposer://tests/vendor/phpunit/phpunit/phpunit:106

testSimplePushAndProcess from laravel-8x-test.DDTrace\Tests\Integrations\Laravel\V8_x\QueueTest (Datadog) (Fix with Cursor)

DDTrace\Tests\Integrations\Laravel\V8_x\QueueTest::testSimplePushAndProcess
Test code or tested code printed unexpected output: spanLinksTraceId: 69a969df000000007dbede904607fcdd
tid: 69a969df00000000
hexProcessTraceId: 7dbede904607fcdd
hexProcessSpanId: 8db73156cf8cf114
processTraceId: 9060924211547602141
processSpanId: 10211684929004957972

View all

ℹ️ Info

❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a65519d | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.11%. Comparing base (a9caaa0) to head (a65519d).
⚠️ Report is 14 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3660      +/-   ##
==========================================
- Coverage   62.21%   62.11%   -0.11%     
==========================================
  Files         141      141              
  Lines       13387    13387              
  Branches     1753     1753              
==========================================
- Hits         8329     8315      -14     
- Misses       4260     4273      +13     
- Partials      798      799       +1     

see 3 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a9caaa0...a65519d. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[pr-commenter]

Benchmarks [ tracer ]

Benchmark execution time: 2026-03-05 12:42:20

Comparing candidate commit a65519d in PR branch add-nightly-system-tests with baseline commit a9caaa0 in branch master.

Found 5 performance improvements and 9 performance regressions! Performance is the same for 180 metrics, 0 unstable metrics.

scenario:MessagePackSerializationBench/benchMessagePackSerialization

• 🟩 execution_time [-4.880µs; -3.940µs] or [-4.501%; -3.634%]

scenario:MessagePackSerializationBench/benchMessagePackSerialization-opcache

• 🟩 execution_time [-8.557µs; -7.023µs] or [-7.589%; -6.229%]

scenario:PDOBench/benchPDOBaseline-opcache

• 🟥 execution_time [+1.791µs; +2.129µs] or [+2.916%; +3.466%]

scenario:PHPRedisBench/benchRedisBaseline

• 🟩 execution_time [-58.269µs; -56.595µs] or [-42.866%; -41.635%]

scenario:PHPRedisBench/benchRedisOverhead

• 🟩 execution_time [-267.416µs; -261.197µs] or [-31.947%; -31.204%]

scenario:SamplingRuleMatchingBench/benchRegexMatching1

• 🟥 execution_time [+93.852ns; +156.748ns] or [+8.123%; +13.567%]

scenario:SamplingRuleMatchingBench/benchRegexMatching1-opcache

• 🟥 execution_time [+86.725ns; +138.875ns] or [+6.807%; +10.900%]

scenario:SamplingRuleMatchingBench/benchRegexMatching2

• 🟥 execution_time [+88.359ns; +150.441ns] or [+7.583%; +12.910%]

scenario:SamplingRuleMatchingBench/benchRegexMatching2-opcache

• 🟥 execution_time [+94.857ns; +129.743ns] or [+7.400%; +10.121%]

scenario:SamplingRuleMatchingBench/benchRegexMatching3

• 🟥 execution_time [+99.530ns; +144.470ns] or [+8.613%; +12.502%]

scenario:SamplingRuleMatchingBench/benchRegexMatching3-opcache

• 🟥 execution_time [+97.659ns; +141.741ns] or [+7.610%; +11.045%]

scenario:SamplingRuleMatchingBench/benchRegexMatching4

• 🟥 execution_time [+90.680ns; +154.920ns] or [+7.831%; +13.379%]

scenario:SamplingRuleMatchingBench/benchRegexMatching4-opcache

• 🟥 execution_time [+102.338ns; +149.062ns] or [+8.010%; +11.666%]

scenario:SymfonyBench/benchSymfonyDdprof-opcache

• 🟩 execution_time [-660.751µs; -325.129µs] or [-7.408%; -3.645%]

[nccatoni]

For this to work two things need to be done:
Adding docker/login-action and nick-fields/retry to the list of authorized actions for the repo
Adding an Datadog API key as a github repo secret

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a65519dc59

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Pin reusable workflow ref before passing API key secrets

This job passes DD_API_KEY/TEST_OPTIMIZATION_API_KEY into a reusable workflow referenced by mutable @main, so any upstream change on DataDog/system-tests immediately executes here with access to those secrets. In the event of an upstream compromise or an unintended change, secrets can be exposed without any commit in this repository; pinning to an immutable commit SHA avoids that supply-chain risk.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Check junit upload exit status before declaring completion

Because this block runs with set +e, datadog-ci junit upload can fail (for example on transient network/auth/API errors) and the script will still continue to print Upload complete. That creates a false-success signal while silently dropping test results from Test Optimization; explicitly checking the upload command status and logging a failure path would prevent this blind spot.

Useful? React with 👍 / 👎.