Mocks don't return useful information when failing parameter filter tests

[fourpastmidnight]

1. General summary of the issue

When using a mock, you can specify a ParameterFilter. When the mock is being evaluated for whether or not it should be executed, a test is performed via Test-ParameterFilter. Unfortunately, this test just returns $true or $false which instructs Pester to not execute the mock and call the real function/cmdlet instead. However, there's no feedback as to why the parameter filter failed. In turn, this leads to less than useful error messages when the test fails.

2. Describe Your Environment

Pester version : 4.4.2 C:\Program Files\WindowsPowerShell\Modules\Pester\4.4.2\Pester.psd1
PowerShell version : 5.1.17134.165
OS version : Microsoft Windows NT 10.0.17134.0

3. Expected Behavior

I expect when I provide a parameter filter like: $Url -eq [Uri]::EscapeUriString($UrlSubstring), that when the parameter filter fails, a useful error message could be provided, such as:

Parameter filter test for mock for {cmdlet/function} with parameters {params} failed.
Expected: {expansion of $Uri}
Actual: {expansion of [Uri]::EscapeUriString(#UrlSubstring)

This would help in determining why one or mocks were not executed for the given parameters, making debugging tests (or code, for that matter) far easier. Instead, I had to troll through the debugger into Mock.ps1, Invoke-Mock, FindMock, and Test-ParameterFilter. On the bright side, I learned how the mocks work, which is quite cool, but most people shouldn't need to know that in order to get some useful information about why their tests are failing.

Also, when using Mock to create a mock, it would be helpful to get a reference to the mock back so we could "inspect" various aspects of it--or, perhaps, allow the ability to create Spies (or interception methods on the mock).

4.Current Behavior

Less than helpful messages are returned when no mocks are executed when verifying mocks.

5. Possible Solution

Add more internal state to the mocks (or the Pester State for the test?) that can be used for reporting back to Pester why a particular mock was not executed.

6. Context

See 1 and 3 above.

[nohwnd]

I was thinking about this few times. What I currently do is that I use Write-Host to write what are the incoming values or how the conditions are evaluated e.g.: Write-Host $Url; Write-Host ([Uri]::EscapeUriString($UrlSubstring)); $Url -eq [Uri]::EscapeUriString($UrlSubstring). This is simpler than debugging.

I don't think that filter failure should report any error, you can have multiple mocks and it would become messy. But it should be possible to add -Verbose parameter to a Mock definition, and Assert-MockCalled and get information about what is happening behind the scenes. E.g. which mocks are considered for use etc.

From what I see most of the problems with mock filters is that people put an object on the left side of -eq and string on the other side which results in $false, because it compares by reference.

I saw someone use Should in the parameter filter because then they got more detailed info, but I don't like it very much and it only works with old version of Pester where Should returned booleans.

So I would start from the verbose output and go from there.

[fourpastmidnight]

Hmm, that’s a good suggestion. I didn’t mean to imply that `Test-ParameterFilter` should return an error, rather, it should return information that could be used when displaying information for mocks which were not called when verifying them. However, I was creating several mocks with varying parameter filters, and so your comment about “things getting messy” is at the forefront of my mind with respect to my proposal. It’s something I’m definitely going to think about more. But as a short term so gap, `Write-Host` seems like a good thing to try. Thanks again for the suggestion.

…

Sent from my Windows 10 phone From: Jakub Jareš Sent: Wednesday, December 12, 2018 17:10 To: pester/Pester Cc: Craig E. Shea; Author Subject: Re: [pester/Pester] Mocks don't return useful information whenfailing parameter filter tests (#1160) I was thinking about this few times. What I currently do is that I use Write-Host to write what are the incoming values or how the conditions are evaluated e.g.: Write-Host $Url; Write-Host ([Uri]::EscapeUriString($UrlSubstring)); $Url -eq [Uri]::EscapeUriString($UrlSubstring). This is simpler than debugging. I don't think that filter failure should report any error, you can have multiple mocks and it would become messy. But it should be possible to add -Verbose parameter to a Mock definition, and Assert-MockCalled and get information about what is happening behind the scenes. E.g. which mocks are considered for use etc. From what I see most of the problems with mock filters is that people put an object on the left side of -eq and string on the other side which results in $false, because it compares by reference. I saw someone use Should in the parameter filter because then they got more detailed info, but I don't like it very much and it only works with old version of Pester where Should returned booleans. So I would start from the verbose output and go from there. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[nohwnd]

Another possibility would be ASTing the script and rewriting the ast to log the evaluation, or replacing the operators with functions automatically and respecting the operator precedence.

nohwnd [1:39 PM]
Theoretically we could parse it and rewrite it to add output to it withotu changing the result, but that would be difficult and I don't want to figure out all the edge cases, nor unbind the scriptblock from the sessionstate.

Joel (discord) APP [1:40 PM]
I have a hunch that doing what you're trying to do is technically possible, but not with a public method. You'd probably need to pull out some reflection and dig into the parser source code.
but I don't really know where to look for this kind of thing, I'm afraid. Parser.cs is a frickin' behemoth

nohwnd [1:49 PM]
My other idea was to replace the operators with functions automatically, to do that I I can just substitute in reverse operator prececdence order, I don't know how it would work when user provides their own parentheses. but this is the basic idea. $i -eq 1 -or $i -eq 3 -> (eq $i 1) -or (eq $i 1) -> (or (eq $i 1) (eq $i 1))

larryweiss [1:52 PM]
@nohwnd This webpage might help
https://social.technet.microsoft.com/wiki/contents/articles/7820.powershell-getting-the-most-from-trace-command.aspx
I think it depends on the set of Tracesources that exist on the machine you are working with.
And now, I am deeper into trying to understand Trace-Command than I have ever been.
social.technet.microsoft.com
PowerShell: Getting the Most from Trace-Command - TechNet Articles - United States (English) - TechNet Wiki
Technical articles, content and resources for IT Professionals working in Microsoft technologies

nohwnd [1:52 PM]
I am sure there is all that I need to do this to be found in Structure and interpretation of computer programs, but writing my own parser is what I want to avoid 😄
Untitled
[string] $pattern = 'consolehost|type'
$debugSources = Get-TraceSource | ?{$_.name -notmatch $pattern } | select -ExpandProperty name

#trace the expression
[scriptblock] $expression = { Get-Process | Select-Object -First 1}
Thanks I tried that yesterday, this uses all trace sources except for the two that are related to console and types because they are ridiculously verbose (maybe the info can be found there? but judging by the names I doubt it).
This prints output for the Get-Process, but not for the predicate I posted. 😞

seeminglyscience [2:56 PM]
@nohwnd rewriting the AST is probably the way I'd go. But you could also traverse the AST and create a column breakpoint for each sequence point

nohwnd [3:00 PM]
@SeeminglyScience Thanks, I want this in a form of "logging to screen" not interactive. Are there some good resources that I should read to learn how to rewrite the AST?

seeminglyscience [3:01 PM]
sorry I meant something like Set-PSBreakpoint -Column 10 -Action { Do-Logging }
similar to code coverage
and no there's no resources, and outside of a few things I've done I'm not sure there are any examples

nohwnd [3:04 PM]
@SeeminglyScience would you be willing to contribute this to Pester? (or at least a concept that I can finish), or is that not interesting to you?

seeminglyscience [3:06 PM]
I can probably do a proof of concept, does pester have any C#? or all PowerShell?

nohwnd [3:08 PM]
there is tiny bit of C# for few types, but mostly powershell, but I know C# so if you do it in C# I can use that or adapt it. We also ship Gherkin .dll so if there is enough benefit from this we can ship it in another .dll if necessary.

seeminglyscience [3:09 PM]
I mainly ask because I already have a class similar to roslyn's SyntaxRewriter but for the PowerShell AST written. So I just need to write an implementation that injects logging of some sort
but that's all in C#

/cc @SeeminglyScience if you consider contributing to this, this is where we can continue the discussion.

[nohwnd]

@SeeminglyScience I forgot to save the link to your AST parsing that you posted to #testing channel. Could you please post it here so I can get back to it when it becomes relevant, please?

[SeeminglyScience]

@nohwnd Here's the demo I posted the other day

Still some things to iron out yet, but it's coming along.

What's the status of this?

[nohwnd]

@rifftual it is not implemented.

[AspenForester]

From what I see most of the problems with mock filters is that people put an object on the left side of -eq and string on the other side which results in $false, because it compares by reference.

Would some more/better Help on this topic reduce such misuses?
If parameter filters aren't working for me, and I feel like I'm doing what the examples suggest, clearly I'm not picking up what the help is putting down. So maybe some additional examples would be in order?