Pester 6.0.0-rc pilot: compatibility signal across 15 real-world v5 repos

[nohwnd]

Why

To gauge how feasible the v5→v6 migration is and get early CI feedback before 6.0.0 final, we opened [DO NOT MERGE] draft PRs pinning ~15 real-world Pester-v5 consumer repos to 6.0.0-rc1/rc2. Each is throwaway (suggested not to merge); the goal was to surface breaks against real suites. This is the aggregated signal.

Overall: 🟢 feasible

Nearly every break is a documented change with a mechanical fix. The only undocumented behavior change found (Should -HaveCount counting dictionary entries) was caught and reverted pre-release. No undocumented break survives. One repo (maester) had a CI cancellation that is not a Pester regression (could not reproduce on rc2).

Pilot PRs

Repo	PR	Signal
dsccommunity/SqlServerDsc	#2464	🟢 green on v6 (after fixes)
dsccommunity/ComputerManagementDsc	#467	🟢 green on v6 (after fixes)
dsccommunity/ActiveDirectoryDsc	#751	🟢 unit green (HQRM red = PSSA, not Pester)
dsccommunity/NetworkingDsc	#545	🟢 unit green
dataplat/dbachecks	#1038	🟢 Linux/Package; Windows pending
Badgerati/Pode	#1732	🟡 documented breaks only
dataplat/dbatools	#10405	🟡 documented breaks only
chocolatey/choco	#3917	🟢 / awaiting approval
chocolatey/cChoco	#194	awaiting CI
Windos/BurntToast	#293	awaiting approval
techthoughts2/Catesta	#108	awaiting approval
PowerShell/PSScriptAnalyzer	#2191	awaiting approval (CLA ok)
maester365/maester	#1884	⚪ inconclusive — not Pester (see below)
igoravl/TfsCmdlets	#281	⚪ not Pester (missing fork secrets)
gaelcolas/Sampler	#577	⚪ Azure didn't trigger
PowerShell/PowerShellEditorServices	—	skipped (no live Pester CI path)

Breaking-change tally (consumer test changes needed)

Change	Documented	Hits	Fix
Mock -ParameterFilter no-match throws (no fall-through to real cmd)	yes	41	add a default Mock, or forward: Mock X { & (Get-Command X -CommandType Cmdlet) @PesterBoundParameters }
Should -HaveCount counts dict entries	no → reverted	40	n/a (reverted in #2783)
Assert-MockCalled removed	yes	8	→ Should -Invoke
Assert-VerifiableMock removed	yes	3	→ Should -InvokeVerifiable
Duplicate AfterAll throws	yes	2	merge blocks
-ForEach $null/@() throws	yes	2	-AllowNullOrEmptyForEach / guard

Lessons

• Mock fall-through removal is the dominant break (41 hits). It's the only change that needs case-by-case work; everything else is mechanical search/replace. Worth highlighting prominently in the migration guide.
• The mechanical breaks (Assert-MockCalled, Assert-VerifiableMock, -ForEach @(), duplicate AfterAll) were trivial to fix and behaved exactly as documented — no surprises.
• One undocumented behavior change slipped in: Should -HaveCount started counting a piped dictionary's entries instead of treating it as one object. The pilot caught it (top breakage bucket) and it was reverted before release.
• maester (CC using Measure-Script #1884): rc1 CI jobs were cancelled at ~3h vs <2min on v5, which looked like a perf regression. Reproduced locally on rc2: full suite finishes in minutes, the 408-file PSScriptAnalyzer.Tests.ps1 runs 661 tests in ~6s with no super-linear discovery cost. Not a Pester regression — likely an env hang (Graph/coverage) or a kill of a queued run. Needs a maintainer re-run on rc2.

Bottom line

Migration is feasible. Breaks are documented and mostly mechanical; the single doc-gap was found and reverted. Suggested follow-ups: emphasize Mock fall-through removal in migration docs, and re-run maester on rc2 to confirm.

[kborowinski]

Do you have any timeframe to implement #2793? As for now I can't run my tests suite in parallel because they require the parameters to be passed by the container. Just asking.

[nohwnd]

Added to 6.0.0 milestone, tasking copilot with it. So either in the initial release, or soon after.

[nohwnd]

Progress update (end of day)

Since filing, two repos we patched with the documented fix recipes have re-run on rc2 — both went green, and Sampler's CI finally triggered. Only changed rows shown:

Repo	This morning	Now	What changed
Badgerati/Pode #1732	🟡 documented breaks only	✅ all green	Applied fix (Assert-MockCalled→Should -Invoke ×151, -ForEach guard ×5). Every build + build-preview leg (mac/ubuntu/win) + snyk pass.
dataplat/dbatools #10405	🟡 documented breaks only	✅ effectively green	Applied fix (Assert-MockCalled→Should -Invoke ×128, New-Object forwarding mock ×9). SINGLE×3 / MULTI / COPY / HADR / RESTART all pass; the one remaining red is Install-DbaInstance failing on WinRM "server name cannot be resolved" — remote-host infra, not Pester.
gaelcolas/Sampler #577	⚪ Azure didn't trigger	🟡 docs-break + PR artifact	CI ran. Most reds = our minimal pilot PR adds no CHANGELOG entry (Should have CHANGELOG.md in the diff) — a PR artifact, not Pester. One genuine Pester red: Get-SamplerAbsolutePath Mock -ParameterFilter no-match throws in Set-SamplerTaskVariable.Tests.ps1 — the documented fall-through removal. Build + coverage pass.

Notes

• The documented fix recipes work end-to-end in production CI. Pode and dbatools were the two largest 🟡 consumers; both are green after mechanical migration plus the New-Object/Mock forwarding-default pattern. Nothing in product code changed.
• Sampler is a nice data point for the dominant break: rc2's diagnostic for the fall-through removal is clear and actionable — "No mock for command 'Get-SamplerAbsolutePath' matched the call … there is no default mock to fall back to. Add a default mock (e.g. Mock Get-SamplerAbsolutePath { ... }) or adjust an existing -ParameterFilter."
• maester CC using Measure-Script #1884 is unchanged — still the original rc1 run cancelled at ~3h (same run/job IDs); needs a maintainer re-run on rc2 to confirm the local result (suite finishes in minutes; not a Pester regression).
• Everything else holds: choco green; dbachecks Linux/Package green with the Windows legs starved for a runner (24h queue timeout, infra); BurntToast / Catesta / PSScriptAnalyzer still gated on first-time-contributor approval.

Net: still zero surviving undocumented breaks, and the two biggest documented-break consumers are now green.

[nohwnd]

@kborowinski rc3 is out with the fix for data passing.

[kborowinski]

@kborowinski rc3 is out with the fix for data passing.

I'll test it tomorrow at work. Thanks a lot!

[kborowinski]

@nohwnd I have an edge case that fails when running in parallel. One of my modules that I use for infrastructure testing KBInfrastructureTesting requires specific version of Pester in the manifest:

RequiredModules = @(
    @{ModuleName = 'KBActiveDirectory'; ModuleVersion = '7.0.0'},
    @{ModuleName = 'KBSystemFunctions'; ModuleVersion = '7.0.0'},
    @{ModuleName = 'KBSystemManagement'; ModuleVersion = '7.0.0'},
    @{ModuleName = 'KBServiceManagement'; ModuleVersion = '7.0.0'},
    @{ModuleName = 'KBDomainComputerManagement'; ModuleVersion = '7.0.0'},
    @{ModuleName = 'Pester'; ModuleVersion = '5.7.1'},
    @{ModuleName = 'PowerShellHumanizer'; ModuleVersion = '13.2.1'},
    'SQLServer',
    'Microsoft.PowerShell.ThreadJob'
)

In one of my tests I'm testing module import, and this fails now:

Context 'Module Import' {
    It '[<_.Name>] Module imports successfully' {
        {Import-Module -ModuleInfo $_ -Force -Verbose:$false -ErrorAction Stop} |
            Should -Not -Throw
    }
}

The error:

[Bukem@VOYAGER]≥# gmo KBInfrastructureTesting -ListAvailable | Test-KBModule -Tag Manifest
Running tests from 1 files in parallel.
[-] Manifest Specification In Module: "KBInfrastructureTesting".Module Integrity.Manifest Property.Module Import.[KBInfrastructureTesting] Module imports successfully 2.87s
 Expected no exception to be thrown, but an exception "The required module 'Pester' was not loaded because no valid module file was found in any module directory." was thrown from D:\Users\Bukem\Documents\PowerShell\.projects\Modules\KBDevelopmentTools\Integration\01-Manifest.Tests.ps1:236 char:22
     + …            {Import-Module -ModuleInfo $_ -Force -Verbose:$false -Erro …
     +               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.
Tests completed in 4.42s
Tests Passed: 21, Failed: 1, Skipped: 0, Inconclusive: 0, NotRun: 0
Exception: Pester run failed, because 1 test failed

The fix is easy. I could move the Pester module specification from RequiredModules to ExternalModuleDependecies and then check for the required version in the root module. But I just wanted to let you know anyways.

[kborowinski]

I've deleted comment with other edge case because that was an actual bug in my module that Run.Parallel exposed 😅