-
-
Notifications
You must be signed in to change notification settings - Fork 3k
Ensure subtest's context kwargs are JSON serializable #13963
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
nicoddemus
wants to merge
3
commits into
pytest-dev:main
Choose a base branch
from
nicoddemus:xdist-subtests-serialization
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+53
−4
Open
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| Fixed subtests running with `pytest-xdist <https://github.com/pytest-dev/pytest>`__ when their contexts contain non-standard objects. | ||
|
|
||
| Fixes `pytest-dev/pytest-xdist#1273 <https://github.com/pytest-dev/pytest-xdist/issues/1273>`__. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO we should strive to have the property that
from_json(to_json(report)) == report. In this case it means that we should haveself.kwargsitself should be serializable, not just into_json.Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree this is desirable, but I don't see how we can accomplish this in a general manner, for any type of object.
In
unittestsubtests,kwargsis permitted to contain any type of object, because it is not serialized to anything and used only for reporting.I don't see how we can support that in functions that serialize to JSON.
Perhaps I'm missing something, could you elaborate a bit?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What I'm thinking is that
kwargsitself contain strings, not the objects themselves, that would of course make it serializable. It is not exposed in the API so should be fine? Unless the original objects are needed for some reason?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well once they are serialized and come back unserialized on the other side, unfortunately they might not be useful depending on the intent. For reporting this should be fine, but if a plugin is using the serialization hooks and expecting the actual objects to be returned this would be a problem.
We can perhaps try to keep the concrete types for objects we know are safe (
int,float, etc.), but for other objects I don't see another solution.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Implemented that, take a look.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, I defer to your judgment on the representation.
However, I still think that
from_json(to_json(report)) == reportis important.If the report itself keeps the kwargs as
dict[str, json-serializable](i.e. performs the conversion in init, not into_serializable), then the property holds. Is there a need to keep the "raw" objects?Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should keep the original objects, because doesn't seem right to me for us to arbitrarily convert the values to something else during
__init__just on the off-chanceto_serializablemight be called eventually. We lose data this way. Nothing prevents a custom plugin to expect the report objects to be intact in their ownpytest_runtest_logreportimplementation.But something occurred to me: there's nothing in
pytest_report_to_serializablethat says the data returned needs to be JSON-native, only JSON-compatible. This means we can convert the data in the report in whenever format we want, as long as the resulting data can be converted into JSON.With this in mind, I think we can simply use
pickleto convert thekwargsinto astr, which we can then easily deserialize duringpytest_report_from_serializable. This will hold thefrom_serializable(to_serializable(report)) == reportproperty, and should support most data types.picklehas its problems, but is the standard way to serialize/deserialize data in Python.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done, please take a look.
One downside that did not occur to me is that local objects (such as
MyEnumused in the test) normally cannot be pickled. I don't think this will be a problem in production, however.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I somewhat think that an arbitrary Python value would be harder for a plugin to handle than a JSON-native value. AFAIK the main/intended use case for the
pytest_runtest_logreporthook is to report the outcome to some other place (terminal, junitxml, etc.) and so what the plugin mostly cares about is the nodeid and outcome and metadata. In case of subtests, I'm not sure how many plugins would want to do something special with the kwargs formatting beyondrepring it.Regarding pickle, I never really used it, but I assumed it wasn't used in pytest for a reason. E.g. execnet doesn't use it for serialization, and pytest also went with json-like for report serialization. Maybe because it's Python specific, or binary, or has multiple versions, or not secure? For this reason my vague sense is that we should avoid it but I don't have a concrete argument.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Plugins are supposed to be "looking" at the value returned by
pytest_report_to_serializable, the value is only meant to be JSON serializable so plugins have some guidance on how it is supposed to be implemented. Ultimately, plugins are expected to pass the returned value over topytest_report_from_serializableand then use the returned object.Perhaps, but I wouldn't just assume that: there is nothing in the docs or the design that says that we will convert the kwargs using saferepr (and lose the original information). I can envision a plugin saving that data to some other logging system, and attempting to recover it later using
pytest_report_from_serializable. If we convert things using saferepr from under it we lose the original metadata.The reasons are:
__init__of some classes if they are pickled,__setstate__, etc). But ultimately we are executing custom Python code anyway directly (the hooks), so this point is moot (we are already executing arbitrary code).I think we are running in circles here:
from_json(to_json(kwargs)) == kwargsand I agree with that.safereprwhen creating* the SubTestReport, which I disagree with: we shouldn't be changing the data like that just because eventually we might call a separate hook for serialization.Perhaps @RonnyPfannschmidt @Pierre-Sassoulas @The-Compiler want to chime in and have other suggestions or share their opinions on what a correct approach would look like?