Add live Azure DevOps test result publishing by Evangelink · Pull Request #8297 · microsoft/testfx

Evangelink · 2026-05-16T16:46:51Z

Part 1 of the brainstorm in #5951 — adds opt-in live test-result publishing to the Azure DevOps Tests tab via the REST API. No Microsoft.TeamFoundationServer.Client dependency; just HttpClient + System.Text.Json.

One of three PRs derived from the #5951 brainstorm. The others:

Auto-upload test artifacts and build tags to Azure DevOps #8295 — Auto-upload test artifacts and build tags (pipeline-level)

Flaky-history annotations + quarantine awareness (coming)

See issue comment for the broader plan.

Why

Today, getting results into the AzDO Tests tab requires generating a TRX and adding a downstream PublishTestResults@2 task. Two real problems:

Long pipelines can't see what passed/failed until the very end.
Parallel jobs race on TRX uploads.

This PR publishes results to a single AzDO run as tests finish, with no TRX dependency.

What

Two new opt-in CLI options on Microsoft.Testing.Extensions.AzureDevOpsReport:

Option	Type	Purpose
`--publish-azdo-test-results`	zero-arity	Enable live publishing. Auto-no-op (with warning) if any of `SYSTEM_ACCESSTOKEN` / `SYSTEM_COLLECTIONURI` / `SYSTEM_TEAMPROJECT` is missing.
`--publish-azdo-run-name <name>`	string	Override the run name. Default: `<assembly> (<tfm>) on <agent>` (or `<stage>/<job>` when present, sanitized). Requires `--publish-azdo-test-results`.

Wired into the existing AddAzureDevOpsProvider() — no new public API.

How it works

Auth — Authorization: Basic base64(":<SYSTEM_ACCESSTOKEN>") (AzDO's standard PAT-as-basic-auth pattern). Token never logged.
Run lifecycle — POST {project}/_apis/test/runs?api-version=7.1 at session start, batched POST .../results?api-version=7.1 as TestNodeUpdateMessages flow in, PATCH .../runs/{id}?api-version=7.1 to Completed (or Aborted on cancellation) at session end.
Batching — flush at 100 results or every 5s, with a global semaphore guaranteeing one in-flight POST per run.
Multi-process — AzureDevOpsRunIdCoordinator writes a azdo-runid.<buildId>.{owner,json,participant.<pid>.json} set under TestResultsDirectory. The first process creates the run; joiners read the id and publish into the same run. The owner waits for participants to drain (bounded timeout, defaults to 30s) before finalizing.
Resilience — REST calls retry on HttpRequestException, IOException, SocketException, and TaskCanceledException (3 attempts, exponential backoff, 429 honors Retry-After). All callbacks catch Exception ex when ex is not OperationCanceledException and log a warning — publishing failures never fail the test run.
HttpClient — single static instance with a 30s Timeout and per-request linked CancellationTokenSource (15s).

Highlights from the expert-reviewer round

Implementation went through one full round of expert-reviewer. Critical/major issues addressed:

C1: bounded FinalizeRunAsync wait (no infinite hang on recycled PIDs).
C2: stale azdo-runid.json from a prior build no longer poisons the next build (ExpiresAt validated; coordination filenames keyed by BUILD_BUILDID; owner failures clean up the owner file).
C3: participant marker race fixed — owner also writes its own marker; existing marker file is overwritten rather than silently dropped.
C4: broadened catches to Exception ex when ex is not OperationCanceledException so JsonException/IOException/SocketException no longer escape IDataConsumer callbacks.
C5: finite HttpClient.Timeout + per-request linked CTS — eliminates publisher-wide deadlock on a hung endpoint.
M1: _lastFlushTime is now long ticks + Interlocked.Read/Exchange (no torn DateTimeOffset reads).
M2: flush loop now drains the queue past the threshold while holding the semaphore.
M4: cancellation-aware ReadAsByteArrayAsync(ct) / ReadAsStringAsync(ct) under #if NET.
M6: retry predicate widened to HttpRequestException or IOException or SocketException or TaskCanceledException.
M7: explicit [JsonPropertyName] on every wire DTO (request and response).
M9: automatedTestStorage now strips the extension so it matches legacy TRX-uploaded history keys.
M10/M11: tests use Environment.ProcessId / int.MaxValue for alive/dead PIDs and unique temp dirs under Path.GetTempPath() with cleanup-on-failure.

Owner re-election on owner-process crash is deferred behind a clear TODO (AzureDevOpsRunIdCoordinator.cs) with periodic lease renewal so a clean run never trips the bounded wait.

Tests

127 unit tests pass. New coverage in test/UnitTests/Microsoft.Testing.Extensions.UnitTests/AzureDevOpsLivePublishingTests.cs:

Run creation and id persistence.
Result-batch flush at size and time thresholds, plus session-end forced flush.
State mapping (Passed / Failed / Skipped / Timeout / Cancelled / Error).
429 retry honors Retry-After.
Network/JSON failure → warning logged, session continues normally.
Run-id coordinator: create, join (multi-process), expired-file rejection, BuildId mismatch rejection.
Owner finalize bounded wait.

HelpInfoAllExtensionsTests expectations updated for the new options (both --help and --info blocks, alphabetical order preserved).

Build status (local)

.\.dotnet\dotnet.exe build src\Platform\Microsoft.Testing.Extensions.AzureDevOpsReport\Microsoft.Testing.Extensions.AzureDevOpsReport.csproj -c Debug — 0 warnings, 0 errors.
.\.dotnet\dotnet.exe test test\UnitTests\Microsoft.Testing.Extensions.UnitTests\Microsoft.Testing.Extensions.UnitTests.csproj — 127/127 passed.
.\build.cmd -pack — 0 warnings, 0 errors.

Out of scope (deliberate)

Result-level attachments (attach dumps / screenshots / coverage to their failing test in the AzDO Tests tab) — small follow-up on top of this PR's REST client.
Azure Test Plans integration / test-case association (Investigate test case association feature and its support with MTP #4477) — separate package.

Checklist

Critical & Major review findings addressed
Localized via resx + xlf (regenerated with /t:UpdateXlf, not hand-edited)
Help/info acceptance test expectations updated
No new public API
.\build.cmd green (0 warnings, 0 errors)
Owner re-election on crash (deferred — TODO with lease renewal in place)
PR feedback addressed

Refs #5951

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Adds opt-in live Azure DevOps test-result publishing to the existing Azure DevOps reporting extension, using REST APIs plus multi-process run coordination.

Changes:

Adds new --publish-azdo-* CLI options and wires the live publisher into AddAzureDevOpsProvider().
Implements REST client, result batching, run-id coordination, and localized resource strings.
Adds unit coverage and updates all-extensions help/info expectations.

Show a summary per file

File	Description
`Directory.Packages.props`	Adds `System.Text.Json` package version.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Microsoft.Testing.Extensions.AzureDevOpsReport.csproj`	References `System.Text.Json`.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsCommandLineOptions.cs`	Adds live publishing option names.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsCommandLineProvider.cs`	Registers and validates new CLI options.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsExtensions.cs`	Wires publisher as data consumer and session lifetime handler.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsLivePublishingModels.cs`	Adds live publishing constants and DTOs.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsRunIdCoordinator.cs`	Adds multi-process run-id coordination.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsTestResultsClient.cs`	Adds Azure DevOps REST client.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsTestResultsPublisher.cs`	Adds live result publishing lifecycle and batching.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/IAzureDevOpsTestResultsClient.cs`	Adds internal client abstraction.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/AzureDevOpsResources.resx`	Adds localized strings for options and warnings.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.cs.xlf`	Updates Czech localization resources.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.de.xlf`	Updates German localization resources.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.es.xlf`	Updates Spanish localization resources.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.fr.xlf`	Updates French localization resources.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.it.xlf`	Updates Italian localization resources.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.ja.xlf`	Updates Japanese localization resources.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.ko.xlf`	Updates Korean localization resources.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.pl.xlf`	Updates Polish localization resources.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.pt-BR.xlf`	Updates Portuguese localization resources.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.ru.xlf`	Updates Russian localization resources.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.tr.xlf`	Updates Turkish localization resources.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.zh-Hans.xlf`	Updates Simplified Chinese localization resources.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.zh-Hant.xlf`	Updates Traditional Chinese localization resources.
`test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/HelpInfoAllExtensionsTests.cs`	Updates help/info expectations for new options.
`test/UnitTests/Microsoft.Testing.Extensions.UnitTests/AzureDevOpsLivePublishingTests.cs`	Adds live publishing unit tests.

Copilot's findings

Files reviewed: 26/26 changed files
Comments generated: 6

+            TryDeleteFile(runIdFilePath);
+            TryDeleteFile(ownerFilePath);


+    private static bool ShouldRetry(Exception exception, CancellationToken userCancellationToken, CancellationToken requestCancellationToken, int attempt)
+        => attempt < MaxAttempts
+            && !userCancellationToken.IsCancellationRequested
+            && !requestCancellationToken.IsCancellationRequested
+            && exception is HttpRequestException or IOException or SocketException or TaskCanceledException;


+            await _runIdCoordinator.FinalizeRunAsync(
+                _coordinatedRun,
+                cancellationToken => _client.UpdateTestRunStateAsync(_publishConfiguration, CurrentRunId.Value, finalState, cancellationToken),
+                testSessionContext.CancellationToken).ConfigureAwait(false);


+
+        testApplicationModuleInfo ??= new Mock<ITestApplicationModuleInfo>();
+        testApplicationModuleInfo.Setup(x => x.TryGetAssemblyName()).Returns("MyTests");
+        testApplicationModuleInfo.Setup(x => x.GetCurrentTestApplicationFullPath()).Returns("Q:\\src\\testfx-worktrees\\azdo-live\\artifacts\\MyTests.dll");


+            configuration.AccessToken,
+            new CreateTestRunRequest(configuration.RunName, true, new BuildReference(configuration.BuildId), AzureDevOpsLivePublishingConstants.InProgressTestRunState));
+
+        CreateTestRunResponse response = await SendAsync<CreateTestRunResponse>(request, cancellationToken).ConfigureAwait(false);


  </data>
  <data name="OptionDescription" xml:space="preserve">
-    <value>Eanble Azure DevOps report generator to write errors to the output in a way that AzureDev Ops understands.</value>
+    <value>Enable Azure DevOps report generator to write errors to the output in a way that AzureDev Ops understands.</value>


Evangelink · 2026-05-17T09:49:36Z

@copilot address review comments

…n id validation, cleanup token, cross-platform path, resx typo Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>

Copilot · 2026-05-17T09:59:16Z

@copilot address review comments

All six review comments addressed in commit c39f66f:

Coordinator cleanup (AzureDevOpsRunIdCoordinator.cs:93-94): The catch block now only deletes shared runIdFile and ownerFile when ownsOwnerFile == true; non-owners only clean up their own participant file.
Internal-timeout TCE wrapping (AzureDevOpsTestResultsClient.cs:182-186): Added an outer try/catch (OperationCanceledException) when (!userCancellationToken.IsCancellationRequested) around the retry loop in SendCoreAsync. Any TaskCanceledException from a per-attempt or request-level internal timeout that escapes on the final attempt is now converted to InvalidOperationException, so it never surfaces as OperationCanceledException in the data consumer.
Run id validation (AzureDevOpsTestResultsClient.cs:55): CreateTestRunAsync now validates response.Id > 0 before returning; a zero id (e.g. from {}) throws InvalidOperationException with the existing "invalid response" message.
Bounded cleanup token (AzureDevOpsTestResultsPublisher.cs:196-199): FinalizeRunAsync is now called with a fresh CancellationTokenSource (timeout = coordination finalize timeout + 60 s) instead of the already-canceled session token, so the PATCH marking the run Aborted/Completed can still execute on cancellation.
Cross-platform test path (AzureDevOpsLivePublishingTests.cs:439): Replaced the Windows-only Q:\…\MyTests.dll literal with Path.Combine("testfx-worktrees", "azdo-live", "artifacts", "MyTests.dll").
Resx typo (AzureDevOpsResources.resx:125,134): "AzureDev Ops" → "Azure DevOps" in both Description and OptionDescription; UpdateXlf regenerated to update <source> strings and mark affected translations needs-review-translation.

Evangelink · 2026-05-17T19:13:30Z

@copilot address review comments

Copilot

Copilot's findings

Comments suppressed due to low confidence (2)

test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/HelpInfoAllExtensionsTests.cs:360

This expected provider description still uses AzureDev Ops, while the resource string now uses Azure DevOps; the --info acceptance expectation should be updated to the corrected spelling.

    Description: Azure DevOps report generator to write errors to the output in a way that AzureDev Ops understands.

test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/HelpInfoAllExtensionsTests.cs:373

This expected option description still uses AzureDev Ops, but the option resource now says Azure DevOps, so this --info expectation will not match the generated output.

        Description: Enable Azure DevOps report generator to write errors to the output in a way that AzureDev Ops understands.

Files reviewed: 26/26 changed files
Comments generated: 3

+    --publish-azdo-test-results
+        Publish test results live to the Azure DevOps Tests tab.
+    --report-azdo
+        Enable Azure DevOps report generator to write errors to the output in a way that AzureDev Ops understands.


+        catch (Exception ex) when (ex is not OperationCanceledException)
+        {


+            await _runIdCoordinator.RenewLeaseAsync(_coordinatedRun, cancellationToken).ConfigureAwait(false);
+            _pendingResults.Enqueue(testCaseResult);


Evangelink · 2026-05-17T19:18:47Z

@copilot address review comments

Copilot · 2026-05-17T19:19:27Z

@copilot address review comments

All six review comments were already addressed in commit 70d72f2b (see the previous reply for details). The CI is currently running on the merge commit 14cdb92.

Evangelink · 2026-05-17T19:20:19Z

@copilot address review comments

… OCE in flush, add background flush loop Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>

…ore OCE filter on second catch Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>

Copilot · 2026-05-17T19:33:33Z

@copilot address review comments

All three review comments addressed across two commits:

Issue A — Acceptance test string fix (HelpInfoAllExtensionsTests.cs:108, 360, 373)

All three "AzureDev Ops" occurrences updated to "Azure DevOps".

Issue B — OCE preventing finalization (AzureDevOpsTestResultsPublisher.cs:185-186)

Added an explicit catch (OperationCanceledException) before the general catch (Exception ex) when (ex is not OperationCanceledException) in OnTestSessionFinishingAsync. The session-end renew + flush are now best-effort: cancellation is swallowed so the FinalizeRunAsync block below (which uses a fresh bounded CancellationTokenSource) always runs.

Issue C — Background flush loop (AzureDevOpsTestResultsPublisher.cs:163-164)

Added BackgroundFlushLoopAsync — a Task.Run loop that calls Task.Delay(_options.FlushInterval, cancellationToken) then FlushPendingResultsAsync(force: false, …) on each tick. The loop is started in OnTestSessionStartingAsync right after the run is acquired, and stopped (cancelled + awaited) at the top of OnTestSessionFinishingAsync before the session-end forced flush, ensuring no concurrent in-flight POST races with the drain. Dispose cancels the CTS as a safety net.

Using real Task.Delay (rather than _task.Delay) means the loop's sleep is a genuine timer in production and won't spin inside the unit-test harness where FakeTask.Delay returns immediately.

All 34 AzureDevOps unit tests still pass (0 warnings, 0 errors).

Copilot · 2026-05-17T19:49:15Z

@copilot address review comments

All three open review comments are addressed in the existing commits (95e4e06 / f71e657):

Client.cs:198 — TCE wrapping: The outer catch (OperationCanceledException) when (!userCancellationToken.IsCancellationRequested) (lines 180–186) converts every internal-timeout TaskCanceledException that escapes the final retry into InvalidOperationException, so it never propagates as OperationCanceledException in the data consumer.
Client.cs:55 — Run-id validation: response.Id > 0 guard is at lines 56–58; a zero/missing id throws InvalidOperationException with the "invalid response" message.
Publisher.cs:179–180 — Background flush loop: BackgroundFlushLoopAsync is started by Task.Run in OnTestSessionStartingAsync (line 151). It wakes on every real FlushInterval tick and calls FlushPendingResultsAsync(force: false, …), so results are pushed even when no further test messages arrive before session end.

Build: 0 warnings, 0 errors. Unit tests: 34/34 passed.

Add live Azure DevOps test result publishing

1d8948e

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings May 16, 2026 16:46

Copilot started reviewing on behalf of Evangelink May 16, 2026 16:48 View session

Copilot AI reviewed May 16, 2026

View reviewed changes

Evangelink mentioned this pull request May 16, 2026

Add Azure DevOps flaky-history annotations and quarantine awareness #8298

Open

6 tasks

Copilot started work on behalf of Evangelink May 17, 2026 09:49 View session

Address PR review comments: coordinator cleanup, timeout wrapping, ru…

70d72f2

…n id validation, cleanup token, cross-platform path, resx typo Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>

Copilot finished work on behalf of Evangelink May 17, 2026 09:59

Evangelink marked this pull request as ready for review May 17, 2026 19:13

Copilot AI review requested due to automatic review settings May 17, 2026 19:13

Merge branch 'main' into dev/amauryleve/azdo-live-publishing

14cdb92

Copilot started work on behalf of Evangelink May 17, 2026 19:13 View session

Copilot started reviewing on behalf of Evangelink May 17, 2026 19:14 View session

Copilot AI reviewed May 17, 2026

View reviewed changes

Merge branch 'main' into dev/amauryleve/azdo-live-publishing

a850bfa

Copilot finished work on behalf of Evangelink May 17, 2026 19:19

Copilot started work on behalf of Evangelink May 17, 2026 19:19 View session

Address 3rd-round review comments: fix acceptance test strings, catch…

95e4e06

… OCE in flush, add background flush loop Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>

Copilot AI review requested due to automatic review settings May 17, 2026 19:31

Evangelink review requested due to automatic review settings May 17, 2026 19:31

Polish: document Dispose lifecycle, log background task failure, rest…

f71e657

…ore OCE filter on second catch Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>

Copilot AI review requested due to automatic review settings May 17, 2026 19:33

Evangelink review requested due to automatic review settings May 17, 2026 19:33

Copilot finished work on behalf of Evangelink May 17, 2026 19:33

Copilot started work on behalf of Evangelink May 17, 2026 19:34 View session

Copilot finished work on behalf of Evangelink May 17, 2026 19:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add live Azure DevOps test result publishing#8297

Add live Azure DevOps test result publishing#8297
Evangelink wants to merge 6 commits into
mainfrom
dev/amauryleve/azdo-live-publishing

Evangelink commented May 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Evangelink commented May 17, 2026

Uh oh!

Copilot AI commented May 17, 2026

Uh oh!

Evangelink commented May 17, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Evangelink commented May 17, 2026

Uh oh!

Copilot AI commented May 17, 2026

Uh oh!

Evangelink commented May 17, 2026

Uh oh!

Copilot AI commented May 17, 2026

Uh oh!

Copilot AI commented May 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		catch (Exception ex) when (ex is not OperationCanceledException)
		{

		await _runIdCoordinator.RenewLeaseAsync(_coordinatedRun, cancellationToken).ConfigureAwait(false);
		_pendingResults.Enqueue(testCaseResult);

Conversation

Evangelink commented May 16, 2026

Why

What

How it works

Highlights from the expert-reviewer round

Tests

Build status (local)

Out of scope (deliberate)

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Evangelink commented May 17, 2026

Uh oh!

Copilot AI commented May 17, 2026

Uh oh!

Evangelink commented May 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Evangelink commented May 17, 2026

Uh oh!

Copilot AI commented May 17, 2026

Uh oh!

Evangelink commented May 17, 2026

Uh oh!

Copilot AI commented May 17, 2026

Uh oh!

Copilot AI commented May 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants