How to differentiate between error and test failures when using hooks?

[amolgawai]

Hi,
I am running pytest using the api pytest.main. I want to collect the test result in the form of no. of succeeded, failed and error. I am able to find successful and failed tests but can’t figure out how to find no. of errors.
I want to add a hook like
@pytest.hookimpl(hookwrapper=True) def pytest_runtest_makereport(item: Item, call: CallInfo):
How do I find if the result was error or test failure?

is this correct approach for creating a summary or is there a simpler api?

Thanks in advance…

[portyu9]

Hi @amolgawai

Use the report's when value in addition to passed / failed.

pytest creates separate reports for setup, the actual test call, and teardown. A normal assertion/test failure is generally:

report.when == "call" and report.failed

while what pytest displays as an ERROR is normally a setup or teardown failure.

For example:

def pytest_runtest_logreport(report):
    if report.when == "call":
        if report.passed:
            passed += 1
        elif report.failed:
            failed += 1
        elif report.skipped:
            skipped += 1

    elif report.when in ("setup", "teardown") and report.failed:
        errors += 1

The important part is to count normal test outcomes only for when == "call"; otherwise a successful test can give you multiple "passed" reports for its different phases.

pytest.main() itself only gives you the overall pytest exit code. Using reporting hooks is the supported way to obtain per-test outcome information.

Please mark this answer as accepted if it helped, thank you.

[amolgawai]

Thank you, this is helpful.