Make Plugins public

[nohwnd]

Review and improve the api of Pester plugins, to be well factored. Think about how each extension point could be used, and how we could extend the api in the future.

Then expose the plugins publicly, and think about how others could register theirs. Especially when they would like to inject them in the middle of our plugins, or replace ours with theirs. And how they would get config.

[fflaten]

replace ours with theirs. And how they would get config.

The config part is important 👍
We soon (if not already) have configuration options to disable most built-in plugins (only Mock left in 5.3.0?

So third party plugins should rely on their own config-options.

[stinos]

As requested in #2020, just to let you know I'm now using the additionalPlugins to workaround code coverage configuration. It ain't pretty, and perhaps I'm missing something but I cannot find a way to make get StartLine/EndLine/... from Invoke-Pester into Enter-CoverageAnalysis; the legacy parameter set doesn't accept a hashtable for -CodeCoverage (well it does but it turns it into the string "System.Collections.HashTable") and PesterConfiguration.CodeCoverage only has Paths and enforces that to be a string array. So now I'm doing something like this:

$ccPlug = & (Get-Module Pester) {Get-CoveragePlugin}
$ccPlugStart = $ccPlug.RunStart
$ccPlugEnd = $ccPlug.End
$ccPlug.RunStart = {
  param($Context)
  $Context.Configuration['Coverage'] = @{
    Enabled = $True
    OutputFormat = 'JaCoCo'
    OutputPath = (Join-Path $PSScriptRoot 'coverage.xml')
    OutputEncoding = 'UTF8'
    Path = 'MyFile.ps1'
    StartLine = 0
    EndLine = 150
    ExcludeTests = $True
    RecursePaths = $True
    CoveragePercentTarget = 75
    UseBreakpoints = $True
    UseSingleHitBreakpoints = $True
  }
  & $ccPlugStart $Context
}
$ccPlug.End = {
  param($Context)
  & $ccPlugEnd $Context
  # Make pester do the post-processing now.
  $Context.TestRun.Configuration.CodeCoverage.Enabled = $True
}

& (Get-Module Pester) {
  $Script:additionalPlugins = $ccPlug
}

Notes:

• ideally I'd like to specify an array of path/startline/endline and while Enter-CoverageAnalysis supports that since it takes such array, the way Get-CoveragePlugin calls it only supports one single configuration object:
  Enter-CoverageAnalysis -CodeCoverage $config -Logger $logger -UseBreakpoints $config.UseBreakpoints
• https://github.com/pester/Pester/search?q=ExcludeTests is declared but not used anywhere? Get-CoverageInfoFromUserInput only refers to IncludeTests

After writing the above I realize it's perhaps better to report this as a bunch separate issues, but I'm new to Pester so perhaps all of this is known already? Otherwise let me know how to proceed.

[JustinGrote]

For reference, the Pester Tests adapter utilizes this API to report test results in real time so it doesn't have to wait for the final output or interpret text output.
https://github.com/pester/vscode-adapter/blob/ebfe917658cf2a310ea0520d42b57ef54d54fb26/Scripts/PesterInterface.ps1#L60-L80

[glennsarti]

@nohwnd Are you ok with me taking some time to come up with a proposal for this? I only ask, so I don't spend lots of time of something you've already "got cooking"

[nohwnd]

Yeah @glennsarti go ahead. I was recently thinking about having a look here, and was considering if strongly typed objects would help here or if they are disadvantage.

Feel free to make a first draft and discuss it here. I am assuming that plugins would have to define a minimum version of pester they support. But then also what would be the story for extension authors if they wish to support both older, and newer versions of Pester (e.g. we add public plugins in 6.0.0, someone publishes the plugin, but then 6.2.0 would introduce something really interesting for the plugin, but they don't want to ditch the 6.0.0 users just yet, or want to pilot some functionality).

If we don't consider this scenario, then I think it is mostly matter of making sure the objects can grow and eventually shrink (ideally in a new major version) as we deprecate features on the object. Much like we do it with any other objects we have, like PesterConfiguration.

[glennsarti]

Proposal TLDR

Plugin

• A single Pester Plugin will be contained with a single PowerShell Module
• A Plugin must export the function Initialize-PesterPlugin -Configuration <PesterConfiguration> -PesterVersion <String> to indicate it is a Pester Plugin
• The Plugin is responsible for taking a user's configuration options, and apply its defaults. This is not Pester's responsibility

Example Pester Plugin registration response;

@{
  "Name" = "Glenns.Pester.Plugin"
  "Configuration" = @{
    "Enabled = $True
    "Debug" = $False 
  }
  "Start" = {
    Write-Host "This is an example step handler"
  }
  "End" = @{
    1 = {
      Write-Host "This is an example of a versioned step handler"
    }
  }
}

Pester Configuration

• Plugins.AutoRegisterPlugins will default to $False until the plugin feature is stable and can be turned on by default.

Setting	Description	Type	Default
Plugins.AutoRegisterPlugins	When enabled, Pester will search all imported Modules with an exported 'Initialize-PesterPlugin' function and register them for use within Pester. All default Plugins will be included. This setting is ignored if Plugins.RegisterPlugins is set.	bool	$False
Plugins.RegisterPlugins	An ordered list of plugin names that will be registered with Pester. Private Plugins will not be automatically included. See Get-PesterPluginSupport for the list of Private plugin names.	string[]	List of all Private Plugin Names
Plugins.Configuration	A PowerShell Hashtable of configuration options.	Hashtable<String, Hashtable<String, Any>>	@{}

Pester (New Exported Function)

Get-PesterPluginSupport - Returns a PSCustomObject containing all supported Step names, supported versions, all default plugin names, and all reserved plugin names |

Example;

  [PSCustomObject]@{
    DefaultPlugins = @('WriteScreen', 'RSpecObjectDecoratorPlugin' ...)
    ReservedPluginNames = @(
      'WriteScreen',
      'RSpecObjectDecoratorPlugin',
       ...
    )
    Steps = @{
      'PluginRegistrationEnd' = @(1)
      'Start' = @(1)
      'DiscoveryStart' = @(1)
      ...
    }
  }

Plugin API Versioning

• Each Step API request for each will be versioned using a single integer. A change in version number indicates a breaking changes.
• A Plugin can support one or more Step API versions, as can Pester.
• During Plugin Registration, step version negotiation will be resolved using simple set intersection (Highest common shared version number)

Plugin Execution

Internally we will use an Adapter to wrap the Plugin scriptblocks into something that Pester can execute. The Adapter is also responsible to enforcing to the Step API version structure.

---

Introduction

There's a lot of prior art we can draw on for a plugin system. Within the unit testing sphere (rspec, nunit, junit etc.), the larger software engineering (HTTP Middleware, linters (golang etc.)), or PowerShell itself (PSDepends, PSFramework). What we can do is break down the requirements for a Plugin System into three basic categories:

• Discovery
• Configuration
• Execution

There are three "actors" in this system; There is the project/repository/PS Module that is using Pester (Consumer), then the Pester Plugin (Plugin) and finally there is Pester itself (Pester). These three things interact with each other to discover, configure and then execute Pester and the Plugin, with the desired configuration.

It is assumed that a Pester Plugin is contained within a single PowerShell Module. (While there could be more than one per module, but that doesn't seem like it would be common, or something you would even want. From a development point of view, it makes it very difficult to find a unique identifier for multiple plugins per module)

Discovery

This requirement defines how Pester and the Plugin can find each other. There are three different styles we can use.

Dynamic Advertisement

The Plugin advertises itself to the broader PowerShell environment saying "Hey, I'm a Pester Plugin". So when Pester is invoked, it "looks" for all of the advertised plugins and then uses them.

This could be by a common public function exposed by the plugin module e.g. Initialize-PesterPlugin, or perhaps by a predefined module name prefix (e.g. XXXXX.PesterPlugin)

The point being, the Plugin only advertises itself, and doesn't require Pester at all.

Problems;

(Probably Minor) The problem with using "magic" strings, be they a Module Name suffix (.PesterPlugin) or a specific exported function (Initialize-PesterPlugin) is that it's transparent
to the User. And you need to make sure the "magic" is obvious enough at what it does and yet not be so common that someone else would choose the name for a different purpose. Out of these two options, having a "magic" function name feels like the better option as no matter what option is chosen there needs to be some function/cmdlet the Pester calls to actually get the Plugin information and script blocks.

(Major) How do you make Plugin order deterministic? Order by module name? Then you end up with people naming Modules with "aaaaSomethingModule" just to force it to be first. Having to name a Module (or any other ordering property to be honest!) so that a User gets a specific Plugin order seems wrong. That's a Pester configuration problem, not a Plugin Author problem.

(Major) How do you not load private/internal plugins?

Static Advertisement

The plugin explicitly tells Pester that "Hey, I'm a Pester Plugin". For example;

When a plugin module is first imported, it also loads the Pester Module, and calls Register-PesterPlugin -......

The point being, loading a Plugin will automatically activate it for use in Pester.

Problem;

(Minor) What happens if you import the Plugin Module and can't load Pester? or worse, imports the wrong version (With many systems having older Pester versions pre-installed)?

(Major) How do you make Plugin execution deterministic? That would mean that the order you import the Plugin modules into your session dictate their execution within Pester. And you can't easily de-register or alter that order.

(Major) How do you not load private/internal plugins?

Consumer Configuration

This is probably the more "common" way other tools load their plugins, although other test tools aren't also a full-blown shell environment. Other tools use a configuration file, which will define what plugins should be enabled, and their configuration. Pester already supports a Configuration object, so we could extend that to define which Plugins should be used (by say Module Name), and in what order. (Ordering is an option common in middleware style products, where it transforms data as information passes from plugin to plugin). For example;

$PesterPreference.Plugins = @('CodeCoverage','....','XXXXXX.PesterPlugin')

And you may have a default plugin list, which are supported within Pester itself

$PesterPreference.DefaultPlugins or via cmdlet Get-PesterDefaultPlugins

Now, that Pester knows what plugins to use, when Pester first starts, it could find a specific cmdlet within the module and use that to inject the plugin into the Pester execution. e.g. This is not exact code but you get the idea...

$PesterPreference.Plugins | ForEach-Object {
  "$_\Get-PesterPlugin" -PesterVersion ...
}

The point being, the Pester configuration by the Consumer defines what modules to use and expects the modules to be available for use.

Problems;

(Minor?) This is the most deterministic and "correct" solution to specifying what Plugins should be used (or ignored) however it requires more from the User to make sure they configure Pester correctly.

Proposal

While the Consumer based configuration is the more correct solution, it is also the most difficult. So instead it is proposed to default to an "Auto-Register" behaviour where Pester will find the appropriate Plugin Modules. However Consumers can opt-out of this behaviour and explicitly set which Plugins will be registered.

Configuration

This requirement defines how a Plugin can configure itself (Default behaviour), how Pester can configure a plugin and finally how can a Consumer define a plugins configuration.

Resolving a Plugin's Configuration

Currently Pester configuration is performed via the New-PesterConfiguration Cmdlet which returns a PesterConfiguration object, which is implemented via a C# Class. This gives Pester the ability to strictly type the configuration and still allow serialization/deserialization via PowerShell Hashtables. However this causes a problem for Plugins, as their configuration is only known at runtime, not at compile time for Pester.

In fact Pester doesn't even know the Plugin configuration at runtime. Plugins are only "registered" at each invocation Invoke-Pester. Calling New-PesterConfiguration doesn't do that so default plugin configuration isn't available because the Plugins aren't available. So when can Pester and the Plugin resolve the configuration? This will occur during the registration workflow; For example

1. Imagine a Plugin called Glenns.Pester.Plugin. It has a default configuration of:

@{
  "Enabled" = $False
  "Debug" = $False 
}

2. Consumer creates a PesterConfiguration and sets an explicit configuration value that it would like for the plugin;

$config = New-PesterConfiguration
$config.Plugins."Glenns.Pester.Plugin" = @{
    "Enabled" = $True
}

Invoke-Pester -Configuration $config
...

3. When Pester registers the public plugin it includes the current Pester Configuration that was used in the Invoke-Pester call

During the registration process Pester calls the registration function of the plugin, and includes the current PesterConfiguration;

...

  Initialize-PesterPlugin -Confguration $currentConfig -PesterVersion '6.0.0'

...

4. The Plugin module recieves the registration request, and extracts the "requested" configuration from the PesterConfiguration. It then resolves the configuration by applying this to the default configuration. This configuration is returned back to Pester in the response of the registration function.

So the result of the registration function would be;

@{
  "Name" = "Glenns.Pester.Plugin"
  "Configuration" = @{
    "Enabled = $True    # Explictly set in the configuration
    "Debug = $False     # Default value
  }
  # ...
}

5. Pester receives the result of the registration function and then updates the current configuration with the resolved plugin configuration e.g.

  $currentConfig.Plugins.$PluginName = $plugin.Configuration

So this workflow allows Pester to query each plugin during registration, resolve the plugin configuration and then store it for later use. However it may be useful for plugins to know when plugin configuration is fully resolved, in case it needs to change its behaviour. As an exmaple, A reporting plugin may change whether to enable colours if it detects a CI plugin is enabled (It's a rough example).

To this end, it's proposed to add a new step "PluginRegistrationEnd" which occurs after all Plugin are registered but before Pester actually starts processing (i.e. before Start)

So as a Plugin author, even though the requested configuration will be a "random hashtable of stuff", we can still use PowerShell classes to define the configuration objects, and then use the type casting to ensure the configuration data is valid. While there are known edgecases with PowerShell classes, if we only use them for simple configuration validation, this is safe to do and is "generally" easy to understand

Pester's Plugin Configuration

Pester needs to be configured for Plugin registration. It is proposed to extend the PesterConfiguration Class with;

A new Plugins configuration section will be created;

Setting	Description	Type	Default
Plugins.AutoRegisterPlugins	When enabled, Pester will search all imported Modules with an exported 'Initialize-PesterPlugin' function and register them for use within Pester. All default Plugins will be included. This setting is ignored if Plugins.RegisterPlugins is set.	bool	$False
Plugins.RegisterPlugins	An ordered list of plugin names that will be registered with Pester. Private Plugins will not be automatically included. See Get-PesterPluginSupport for the list of Private plugin names.	string[]	List of all Private Plugin Names
Plugins.Configuration	A PowerShell Hashtable of configuration options.	Hashtable<String, Hashtable<String, Any>>	@{}

Execution

This requirement defines how Pester executes a Plugin, at the correct time within the lifecycle, with the correct configuration. It also defines how Pester responds when a Plugin behaves in an unexpected way.

Pester already has a fairly robust internal plugin execution system. This already runs major components of Pester. It is somewhat exposed for public consumption however it is not officially supported not advised to use it. However it is definitely only designed for internal usage; It exposes inner workings of Pester itself (The Context objects are complex and contain a lot of internal information), has no versioning system or API contracts (In this sense an API contract is the shape and values of the $Context object which is passed during Plugin Step execution)

We will refer to the two different types of Plugins as;

• Private Plugins These are the existing Plugins and are private; They are the within Pester itself and managed with the Pester source code

• Public Plugins These are the proposed Plugins, which are created by authors other than the Pester team. The source code lives outside of Pester.

There are three ways we could introduce a more appropriate way for Pester Plugin authors to use;

1. Rewrite the existing Plugin system to use structured API contracts and then expose that for Plugin Author use.

This is a significant amount of work to replace something that, quite honestly, is already working. These private plugins are for the internal workings of Pester itself. There is little advantage to making them "Public".

2. Add a new Plugin system that runs side-by-side with the existing plugin system (Parallel)

The existing plugin is already proven and handles many of the edgecases users have come across. Having to rework that for two plugin system makes little sense and only end up having two versions of the same execution system, just with different plugin types (Public vs Private)

3. Do not modify the existing plugin system, but instead introduce a Public Plugin adapter. Where Public Plugins are still executed using the existing Plugin system, BUT they are wrapped/adapted so that the calls to the Plugin are structured (according to the API contract) and don't emit Pester internals.

It is proposed to use Method No. 3, using an adapter.

How does this work?

Imagine we have a plugin called AwesomePlugin and has a single step ContainerStart. During plugin discovery it gets registered with Pester;

AwesomePlugin\Get-PesterPlugin -PesterVersion '6.0'

Pester will take that Plugin and then wrap that plugin with an Adapter, and then register THAT within the plugin system so you end up with something like;

+-----------------------------------------------+
|   *PluginAdapter*                             |
|                                               |
|----------------+ ---> +----------+ --------+  |
| ContainerStart |      | Magic 🪄 |         |  |
|----------------+ <--- +----------+ <----+  |  |
|                                         |  |  |
|  +----------------------------------+   |  |  |
|  |   *AwesomePlugin*                |   |  |  |
|  |                 +----------------| --+  |  |
|  |                 | ContainerStart |      |  |
|  |                 +----------------| <----+  |
|  |                                  |         |
|  +----------------------------------+         |
|                                               |
+-----------------------------------------------+

Where the plugin adapter exposes the ContainerStart step. But when it's actually called, there's some magic which then actually calls the AwesomePlugin\ContainerStart step. The magic performs many functions;

• It converts the full $Context object into the API contract, and then is used to call into AwesomePlugin. As a bonus, it means that an adapter can support multiple API contract version (More on that later).

• Unlike the Private Plugins, when calling the ContainerStart step inside the AwesomePlugin it will use a Closure. This will ensure that Pester internals are not exposed, and that each plugin is segregated appropriately Not needed at all. Variables within Modules are scoped to only inside that module by default. Calling a scriptblock from one module, that was created in another module, does not expose any variables.

• Once the call to the AwesomePlugin is complete, the adapter can convert from the API contract back into the $Context object. Allowing plugins to modify data only when allowed.

API Contract Versioning

Initially it was thought that the Pester Version would dictate the API contract version but realistically that's not what really happens. The defining factor here is the version of a Step API version, not the version of Pester. Also, it would be very rare that a new Pester version would cause breaking changes across all steps. Ideally we should be decoupling Plugin API contract versions from Pester versions. For example;

Say Pester 6.0 supports the following;

Pester	Supported
6.0	ContainerStartv1

The next version of Pester then has a new API contract for ContainerStart which is not backwards compatible. So it calls for a new version, 2. So now the support list looks like;

Pester	Supported
6.0	ContainerStartv1
6.1	ContainerStartv2

But, we can actually have better support. The adapter will have the ability to support multiple API contract version if it chooses. So a better support matrix would be;

Pester	Supported
6.0	ContainerStartv1
6.1	ContainerStartv1 and ContainerStartv2

Why is supported multiple version better than only a single one?

In short, because of the migration lag time when releasing API versions. If Pester supported one and only one API version per Step then whenever that API version is changed ALL plugins will fail to register, and ultimately cause Pester to raise exceptions, until the Plugin Authors update their plugin to support the new version. This is a terrible user experience. By being able to support more than one API version then the Pester maintainers can release new API versions, and have a known deprecation schedule. In the example above;

In the example above Pester 6.1 supports both V1 and V2 of ContainerStart. If a Plugin Author uses ContainerStartv1 then pester will raise a Warning saying "Hey, this plugin needs an update" but still continue to function. And then in 6months time when Pester 6.2 is released then the ContainerStartv1 support can be removed.

API Contract Version negotiation

Now that we know Pester can support multiple versions of the same Step API, how does it negotiate with the Plugin to determine which version it uses. This can be achieved during the Plugin registration phase using a fairly simple process;

1. Pester calls the Initialize-PesterPlugin in the Plugin Module.
2. The Plugin Module returns a Hashtable which includes which steps and which version of that step it supports e.g.;

@{
  "ContainerStart" = @{
    1 = { ... Script Block ... }
    2 = { ... Script Block ... }
  }
}

This hash says that it provides both Version 1 and Version 2 of the ContainerStart Step

3. Pester then takes the intersection of versions it supports and versions the Plugin supports. And then takes the highest number. For example;

Pester Support	Plugin Support	Resulting Version
2,3	1,2	2. Both Pester and Plugin support v2.
1,2	1,2	2. Both Pester and Plugin support v1 and v2, but v2 is the highest.
2,3	1	Raises an exception during registration as there are no common supported versions

API Version naming

Ah naming things! The most troubling of all computer science problems. While at first thought using SemVer (Semantic Versioning) strings seemed like the obvious choice but they are typically stored as strings which requires casting. Sorting SemVer numbers is tricky, and ultimately far too complex for what we need. We only need to express breaking changes. These API version numbers don't need the concepts of Minor or Patch changes.

So using integers is much better naming system. Easier to sort, easier to understand as a Plugin Author and requires less "cognitive load".

Pester Lifecycle Steps

Currently Pester supports the following Steps

PluginRegistrationEnd
Start
    DiscoveryStart
        ContainerDiscoveryStart
            BlockDiscoveryStart
                TestDiscoveryStart
                TestDiscoveryEnd
            BlockDiscoveryEnd
        ContainerDiscoveryEnd
    DiscoveryEnd
    RunStart
        ContainerRunStart
            OneTimeBlockSetupStart
            EachBlockSetupStart
                OneTimeTestSetupStart
                    EachTestSetupStart

                    EachTestTeardownEnd
                OneTimeTestTeardownEnd
            EachBlockTeardownEnd
            OneTimeBlockTeardownEnd
        ContainerRunEnd
    RunEnd
End

[glennsarti]

Okay, that's enough draft. Now for a proof of concept

Proof of Concept done. (https://github.com/glennsarti/Pester/tree/tmp-diagnostic-plugin)

[x] Plugin Loading
[x] Plugin Execution by both Invoke-ScriptBlock and Invoke-PluginStep
[x] API Versions detection
[x] Basic debug messages
[x] Using a proper Pester configuration object to hold the loadable Plugins list. For now, just did a shared variable called $PesterPreferencePlugins
[x] Not loading system/private plugins

Didn't do;

[ ] Full plugin configuration
[ ] Public Payload wrapping (Full Context --> Public API context before the Public call and then Public API context back into Full Context after the Public call

POC Module

Diagnostic.Pester.Plugin.psd1

@{
    RootModule        = 'Diagnostic.Pester.Plugin.psm1'

    # Version number of this module.
    ModuleVersion     = '1.0.0'

    # ID used to uniquely identify this module
    GUID              = 'd0e76535-3d7c-4293-b7d9-a6e35bff7052'

    # Description of the functionality provided by this module
    Description       = 'Demo plugin for Pester 6'

    # Functions to export from this module
    FunctionsToExport = @(
        'New-DefaultConfiguration'
        'Get-PesterPlugin'
    )
}

Simple Plugin

Diagnostic.Pester.Plugin.psm1

Function New-DefaultConfiguration {}

Function Get-PesterPlugin {
    @{
        "ContainerDiscoveryStart" = {
            param($Context, $StepName, $StepVersion)
            Write-Host "Inside Step $StepName@$StepVersion"
        }
    }
}

Fancy Plugin

Registers every version of every step and shows how to use shared variables across step invocations.

Function New-DefaultConfiguration {}

Function Get-PesterPlugin {
    $script:Somevalue = 1

    $sb = {
        param($Context, $StepName, $StepVersion)
        Write-Host "Inside Step $StepName@$StepVersion !$script:Somevalue!" -ForegroundColor Cyan
        $script:Somevalue += 1
    }

    $plugin = @{}

    $list = (Get-PesterPluginSupport).Steps
    foreach($StepName in $list.Keys) {
        $scriptBlocks = @{}
        foreach($StepVersion in $list[$StepName]) {
            $scriptBlocks[$StepVersion] = $sb
        }
        $plugin[$StepName] = $scriptBlocks
    }

    Write-Output $plugin
}