Add Microsoft.Testing.Extensions.GitHubActionsReport (groups, annotations, step summary, slow-test notices)#9541
Conversation
…y log groups Implements the first requirement of microsoft#9142: a new GitHub Actions report extension that emits ::group:: / ::endgroup:: workflow commands per test assembly so the runner UI collapses each assembly's output. Auto-activates when GITHUB_ACTIONS=true; controllable via --report-gh-groups on/off and --report-gh. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Emit a ::error workflow command for each failing test via a new GitHubActionsAnnotationReporter (IDataConsumer), mirroring the AzureDevOpsReport structure. Resolves the source file/line from the exception stack trace relative to GITHUB_WORKSPACE (or the .git root), skipping MSTest assertion-implementation frames, and falls back to a title-only annotation when no location is available. Adds the --report-gh-annotations on|off knob (default on) and a shared GitHubActionsFeature activation helper. Includes unit tests for the annotation formatting/escaping and option validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add GitHubActionsSummaryReporter that collects per-test outcomes and, at session end, appends a markdown roll-up (totals table, failures list, slowest tests) to the file named by GITHUB_STEP_SUMMARY, mirroring the AzureDevOpsSummaryReporter structure. Adds the --report-gh-step-summary on|off knob (default on) plus unit tests for the markdown shape. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add GitHubActionsSlowTestReporter that tracks in-progress tests and, on a 1s scan loop with exponential backoff, emits a ::notice workflow command for any test still running past a threshold (default 60s), mirroring the AzureDevOpsSlowTestReporter. Adds the --report-gh-slow-test-notices on|off knob (default on) and a --report-gh-slow-test-threshold seconds option, plus deterministic ScanOnce-driven unit tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Update PACKAGE.md to describe failure annotations, the job summary, and slow-test notices alongside log groups, with a table of all --report-gh options. Minor test cleanup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
In CI the terminal output device uses SimpleAnsi mode and emits a color
reset ('ESC[m') without a trailing newline after the colored failed-test
block. The failure annotation, emitted immediately after via the same
output device, inherited that dangling reset as a line prefix ('ESC[m::error
...'). GitHub only recognizes a workflow command when the line begins with
'::', so the annotation was silently dropped (groups and notices are emitted
at clean line boundaries and were unaffected).
Emit the annotation on a fresh line so '::error' always starts at column 0.
Verified end-to-end via the gh-report-validation harness: the failure
annotation now surfaces in the GitHub Actions Annotations panel and on the
file's diff gutter.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The property was set twice in the same PropertyGroup; keep the single copy with the descriptive comment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds a new Microsoft.Testing.Extensions.GitHubActionsReport extension (parallel to AzureDevOpsReport) that emits GitHub Actions workflow commands to improve CI UX (log grouping, ::error annotations, step summary markdown, and slow-test ::notice messages). It also wires the extension into the solution and unit-test project, and updates packaging behavior in TestFramework.Extensions.
Changes:
- Introduces the new GitHub Actions report extension package (reporters, command-line options, resources, packing hooks, PublicAPI).
- Adds unit tests covering escaping, CLI validation, group emission, annotations, markdown summary shape, and slow-test notices.
- Updates solution/project wiring and adds pack-time adjustment logic for analyzer dependency handling.
Reviewed changes
Copilot reviewed 45 out of 45 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| TestFx.slnx | Adds the new GitHub Actions extension project to the solution. |
| test/UnitTests/Microsoft.Testing.Extensions.UnitTests/Microsoft.Testing.Extensions.UnitTests.csproj | References the new extension project for unit testing. |
| test/UnitTests/Microsoft.Testing.Extensions.UnitTests/GitHubActionsSummaryReporterTests.cs | Tests markdown step-summary generation (totals, failures, slowest tests). |
| test/UnitTests/Microsoft.Testing.Extensions.UnitTests/GitHubActionsSlowTestReporterTests.cs | Tests slow-test notice formatting, thresholding, and backoff behavior. |
| test/UnitTests/Microsoft.Testing.Extensions.UnitTests/GitHubActionsReporterTests.cs | Tests GitHub Actions group command emission and enablement logic. |
| test/UnitTests/Microsoft.Testing.Extensions.UnitTests/GitHubActionsEscaperTests.cs | Tests escaping rules for workflow command data/properties. |
| test/UnitTests/Microsoft.Testing.Extensions.UnitTests/GitHubActionsCommandLineProviderTests.cs | Tests CLI option validation behavior for on/off and thresholds. |
| test/UnitTests/Microsoft.Testing.Extensions.UnitTests/GitHubActionsAnnotationReporterTests.cs | Tests ::error annotation construction and escaping, with/without source location. |
| src/TestFramework/TestFramework.Extensions/TestFramework.Extensions.csproj | Adds pack-time tweak intended to ensure MSTest analyzers flow to consumers without running during repo build. |
| src/Platform/Microsoft.Testing.Platform/Microsoft.Testing.Platform.csproj | Grants InternalsVisibleTo for the new extension. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Microsoft.Testing.Extensions.GitHubActionsReport.csproj | New extension project definition + packing layout + platform reference. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsExtensions.cs | Registers reporters and CLI provider on the test application builder. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsFeature.cs | Centralizes master/knob enablement rules (env + --report-gh + --report-gh-*). |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsReporter.cs | Emits per-assembly ::group:: / ::endgroup::. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsAnnotationReporter.cs | Emits ::error ...::... failure annotations with optional file/line resolution. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryReporter.cs | Appends step summary markdown to GITHUB_STEP_SUMMARY. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSlowTestReporter.cs | Emits ::notice for long-running tests with exponential backoff. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsEscaper.cs | Implements GitHub workflow-command escaping rules. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsCommandLineProvider.cs | Defines CLI options and validation for the extension’s features. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsCommandLineOptions.cs | Declares option names and defaults. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsRepositoryRoot.cs | Resolves workspace/git root to convert absolute paths into repo-relative annotation paths. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/TestNodeIdentity.cs | Resolves stable test identity (FQN property fallback to DisplayName). |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/TargetFrameworkMonikerHelper.cs | Computes a short TFM/moniker for display. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/TestingPlatformBuilderHook.cs | MSBuild hook entry point for adding the extension at build time. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/buildMultiTargeting/Microsoft.Testing.Extensions.GitHubActionsReport.props | Declares the MSBuild builder hook item for multi-targeting. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/build/Microsoft.Testing.Extensions.GitHubActionsReport.props | Imports multi-targeting props for build/TFM-specific consumption. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/buildTransitive/Microsoft.Testing.Extensions.GitHubActionsReport.props | Imports multi-targeting props transitively. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/BannedSymbols.txt | Enforces testability conventions (IClock/ITask/RoslynString, etc.). |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/PACKAGE.md | Package documentation and CLI options. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/PublicAPI/PublicAPI.Shipped.txt | Initializes shipped public API baseline. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/PublicAPI/PublicAPI.Unshipped.txt | Declares new public API surface for the extension. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/GitHubActionsResources.resx | Adds localized resource strings for CLI/help/error messages. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.cs.xlf | Generated XLF for localization. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.de.xlf | Generated XLF for localization. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.es.xlf | Generated XLF for localization. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.fr.xlf | Generated XLF for localization. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.it.xlf | Generated XLF for localization. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.ja.xlf | Generated XLF for localization. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.ko.xlf | Generated XLF for localization. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.pl.xlf | Generated XLF for localization. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.pt-BR.xlf | Generated XLF for localization. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.ru.xlf | Generated XLF for localization. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.tr.xlf | Generated XLF for localization. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.zh-Hans.xlf | Generated XLF for localization. |
| src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.zh-Hant.xlf | Generated XLF for localization. |
| <Target Name="_UnprivatizeAnalyzersForPack" BeforeTargets="_GetPackageDependencies"> | ||
| <ItemGroup> | ||
| <PackageReference Update="MSTest.Analyzers" PrivateAssets="none" /> | ||
| </ItemGroup> | ||
| </Target> |
…mpositeExtensionFactory The log-group GitHubActionsReporter is only registered under one host hook (AddTestSessionLifetimeHandler), so wrapping it in a CompositeExtensionFactory provided no benefit -- that factory only matters for extensions registered under multiple hooks that must share a single instance (the summary and slow-test reporters). Register it directly, matching the GitHubActionsAnnotationReporter and the AzureDevOpsReporter precedent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file's MSTest.Analyzers PackageReference + _UnprivatizeAnalyzersForPack target were an accidental leftover unrelated to the GitHub Actions extension. They duplicated the existing MSTest.Analyzers.Package ProjectReference already present upstream (which correctly flows analyzer/build assets to consumers via PrivateAssets=none ExcludeAssets=compile;runtime), and the added block was buggy (ExcludeAssets=all would leak into the nuspec and block analyzer flow). Restore the file to match microsoft/main. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Review remarksNice work — escaping, slow-test threading, and localization all look solid. Collecting the remaining points from my pass below (the multi-assembly Major1. No help/info acceptance coverage for the new options 2.
An exception here propagates into the platform's data-consumer dispatch instead of degrading to "no annotation for this test". Please mirror the sibling consumers' try/catch + Moderate3. 4. 5. Brittle, checkout-coupled annotation test assertions 6. Not wired into the MSTest.Sdk 7. Group Low / nit8. 9. No 10. 11. AzDo links |
Register the Microsoft.Testing.Extensions.GitHubActionsReport package in the AllExtensionsInfoTest fixture and add its six --report-gh* options to both the --help and --info expected output blocks in HelpInfoAllExtensionsTests. Verified with build.cmd -pack + the acceptance test (9/9 passing). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Wire Microsoft.Testing.Extensions.GitHubActionsReport into the MSTest.Sdk AllMicrosoft profile, mirroring AzureDevOpsReport: default-enable property + conditional PackageReference/PackageVersion in ClassicEngine.targets, a VSTest unsupported-option guard, and a NativeAOT unsupported warning. Extend SdkTests with a per-extension selectively-enabled row and add --report-gh to the EnableAll command line. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ared helpers The GitHubActionsReport extension carried its own copies of TargetFrameworkMonikerHelper and TestNodeIdentity that duplicated logic already present in (or alongside) SharedExtensionHelpers. Merge them into src/Platform/SharedExtensionHelpers so both AzureDevOpsReport and GitHubActionsReport share one implementation: - TargetFrameworkMonikerHelper: the two divergent methods are merged under distinct names -- GetTargetFrameworkMonikerIncludingPlatform (existing shared behavior) and GetTargetFrameworkMonikerWithRuntimeFallback (former GitHub behavior); all call sites repointed with behavior preserved. - TestNodeIdentity: the two copies were identical, so they collapse to a single shared GetTestName; the AzureDevOpsReport copy is moved to the shared folder and the GitHub copy deleted, with the shared file compile-linked into both extensions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…s-report # Conflicts: # src/Platform/SharedExtensionHelpers/TestNodeIdentity.cs
…perimental and pack as 1.0.0-alpha
Task 1 - Split slow-test reporter into shared + pipeline-specific:
- Extract the common plumbing (in-progress tracking, background scan loop,
exponential backoff, session lifecycle) into
SharedExtensionHelpers/SlowTestReporterBase.cs, linked into both the Azure
DevOps and GitHub Actions report extensions. Each host now only supplies
IsEnabled, GetTestName, threshold resolution, an optional activation gate,
and the host-specific line rendering.
Task 2 - Make the GitHub Actions reporter API experimental:
- Mark GitHubActionsExtensions with [Experimental("TPEXP")] and flag the
matching PublicAPI entries with [TPEXP], mirroring JUnitReport.
Task 3 - Pack Microsoft.Testing.Extensions.GitHubActionsReport as 1.0.0-alpha:
- Add version-prefix/prerelease-label properties (Directory.Build.props),
wire them through the csproj, MSTest.Sdk.csproj template properties and
Sdk.props.template, and drop the CommonVersion default in
ClassicEngine.targets, mirroring JUnitReport.
Merge fix - Resolve TestNodeIdentity cross-assembly ambiguity introduced by
merging main (which added AzureDevOpsTestNodeIdentityTests) with the branch's
shared-namespace TestNodeIdentity consolidation: give GitHubActionsReport its
own TestNodeIdentity so the shared type is not duplicated across two
test-visible assemblies.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…n in acceptance tests The GitHubActionsReport project was present in TestFx.slnx and wired into the MSTest SDK / acceptance tests, but was missing from the solution filters the CI builds and packs from (Microsoft.Testing.Platform.slnf, NonWindowsTests.slnf), so the package was never produced. This caused NU1101 (package not found) in the HelpInfoAllExtensionsTests and MSTestSdk acceptance assets. - Add Microsoft.Testing.Extensions.GitHubActionsReport to both solution filters (alongside AzureDevOpsReport) so it gets built and packed in CI. - Because the package now ships as 1.0.0-alpha (not the platform common version), reference it via a dedicated $MicrosoftTestingExtensionsGitHubActionsReportVersion$ placeholder read from the packed nupkg (mirroring JUnitReport/CtrfReport), instead of $MicrosoftTestingPlatformVersion$. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…crosoft#2) - Wrap per-node work in try/catch + LogWarning, mirroring the sibling GitHubActionsSlowTestReporter/GitHubActionsSummaryReporter so a failure while building/emitting an annotation (e.g. a malformed stack-trace path making IFileSystem.ExistFile throw) degrades to "no annotation for this test" instead of propagating into the platform's data-consumer dispatch. - Use FirstOrDefault instead of SingleOrDefault for the test-node state so a malformed node carrying multiple state properties does not throw. - Drop the dead testDisplayName parameter and compute GetTestName once. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…iew microsoft#7) GitHubActionsReporter previously implemented only ITestSessionLifetimeHandler and emitted ::endgroup:: unconditionally, relying purely on registration order. Mirror the deliberate engineering of AzureDevOpsLogGroupReporter: - Also implement IDataConsumer with a no-op ConsumeAsync so OnTestSessionFinishingAsync runs in the consumer phase (after producer-only handlers), and register it last as a data consumer so the closing ::endgroup:: wraps all other reporters' output. - Add a _groupOpened flag so ::endgroup:: is only emitted when a matching ::group:: was opened, and wrap both lifecycle callbacks in try/catch + LogWarning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…test The known-extension registration acceptance test covered AzureDevOpsReport but not the new GitHubActionsReport, leaving the extension's MSBuild self-registration hook unvalidated. Add it (alphabetically): the package reference + version patch in the generated asset, a --report-gh --help assertion, and an assertion that Microsoft.Testing.Extensions.GitHubActionsReport.TestingPlatformBuilderHook.AddExtensions is emitted into the generated SelfRegisteredExtensions source. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…osoft#5) The two source-location tests hard-coded the physical line numbers of their throw statements (line=19/40) and relied on the real repo checkout + SystemFileSystem, so any edit above the throws or a relocated/packaged layout would break them. - Compute the throw line dynamically via a CaptureException helper + [CallerLineNumber] so edits elsewhere in the file no longer shift a hard-coded expectation. - Inject a mock IFileSystem (every file "exists") so resolution no longer depends on the source file physically being present. - Assert the resolved file suffix / line / escaping instead of a full checkout-coupled absolute path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…icrosoft#9) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…eview microsoft#8) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ft#11) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…iew microsoft#10) The slow-test threshold is validated by the command-line provider and re-read here. Document that this mirrors the sibling AzureDevOpsSlowTestReporter (which reads its options straight from ICommandLineOptions) rather than threading a parsed value through, keeping the two reporters consistent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The extension is now packed as an experimental 1.0.0-alpha package (1.0.0-dev in dev builds) instead of following the platform version, matching the finalized PR microsoft/testfx#9541. Refresh the locally-built packages and bump the extension reference from 2.3.0-dev to 1.0.0-dev.
… requirement The extension now requires BOTH GITHUB_ACTIONS=true and the --report-gh master switch to be enabled. Seed the master switch in the GitHubActionsReporter/SlowTestReporter test helpers so the "enabled" tests reflect the new precondition (the env/knob behavior is exercised on top of it). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…eview microsoft#11) Instead of duplicating the git-root walk, FindGitRoot now calls the shared Microsoft.Testing.TestInfrastructure.RootFinder.Find() (linked into the project, as the Azure DevOps extension already does) and translates its InvalidOperationException into null so a reporter running outside a git checkout still degrades to "no source location". Because RootFinder is now compiled into both the AzureDevOpsReport and GitHubActionsReport assemblies (both InternalsVisibleTo the extensions unit tests), the unqualified RootFinder reference in the GitHub test files becomes ambiguous. Expose the GitHubActionsReport reference under a `ghactions` extern alias and qualify the GitHub test files' usings so the AzureDevOps tests keep binding to their copy. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Summary
Fixes #9142.
Adds a new
Microsoft.Testing.Extensions.GitHubActionsReportextension package (sibling of the existingMicrosoft.Testing.Extensions.AzureDevOpsReport) that emits GitHub Actions-native workflow commands so test runs on GitHub Actions produce a first-class experience. This PR implements all four requirements from the issue.What's included
All four building blocks from #9142:
::group::/::endgroup::so each test assembly's output is collapsed by default in the runner UI (GitHubActionsReporter.cs).::error file=…,line=…,title=…::command for each failing test, so failures appear in the workflow Annotations tab and, when the source location can be resolved, on the pull request's Files changed diff gutter (GitHubActionsAnnotationReporter.cs).GITHUB_STEP_SUMMARY, which GitHub renders on the workflow run summary page (GitHubActionsSummaryReporter.cs).::noticecommand for any test still running past a threshold, default 60s (GitHubActionsSlowTestReporter.cs).Supporting pieces:
GitHubActionsEscaper.cs— escapes special characters per GitHub workflow-command rules.GitHubActionsCommandLineProvider.cs/GitHubActionsCommandLineOptions.cs— the master switch plus per-feature toggles.GitHubActionsRepositoryRoot.cs/TestNodeIdentity.cs/TargetFrameworkMonikerHelper.cs— source-location and identity helpers used for annotations..resx+ regenerated.xlffor all locales),PublicAPI,BannedSymbols,PACKAGE.md, build props.TestFx.slnx, PlatformInternalsVisibleTo, and the unit-test project.Behaviour
GITHUB_ACTIONS=true; otherwise silently no-ops.--report-ghmaster switch.--report-gh--report-gh-groups on|off--report-gh-annotations on|off--report-gh-step-summary on|off--report-gh-slow-test-notices on|off--report-gh-slow-test-threshold <seconds>Tests
New unit tests in
test/UnitTests/Microsoft.Testing.Extensions.UnitTests/:GitHubActionsEscaperTests.csGitHubActionsCommandLineProviderTests.csGitHubActionsReporterTests.cs(group emission)GitHubActionsAnnotationReporterTests.csGitHubActionsSummaryReporterTests.csGitHubActionsSlowTestReporterTests.csRelated
Microsoft.Testing.Extensions.AzureDevOpsReport— sibling extension.