Skip to content

[BUG]: Coverage deploy broken since Aug 15 — .coveragerc omit list enumerates Cython .pxd files by hand #2723

Description

@rparolin

Part of #2722.

The CI: Coverage workflow has failed every night since Aug 15 (last full green: run 31757710811, Aug 14), with a single incidental pass on Aug 23.

Symptom

The "Combine Coverage and Deploy" job fails at the step named "Combine coverage data":

=== Combining coverage data ===
Combined 2 files

=== Generating HTML, XML, and text reports ===
Combined 2 files
Plugin 'Cython.Coverage.Plugin' did not provide a file reporter for 'cuda/bindings/_lib/cyruntime/cyruntime.pxi'.
##[error]Process completed with exit code 1.

On other nights the same error names a different file — e.g. run 32539916399 (Aug 22) and run 32676345017 (Aug 24) both report cuda/core/_memory/_location.pxd.

The step name is misleading. coverage combine succeeds — the log prints Combined 2 files. The failure is coverage html, four commands later in the same run: block, surfacing under the step's name because the shell is bash -e. This cost real time during diagnosis.

Root cause

Cython has two file types that are not standalone modules: .pxd (declaration/header) and .pxi (textually included into another file). Neither compiles to its own translation unit, so Cython.Coverage.Plugin cannot construct a FileReporter for them. When coverage.py is handed one, the plugin returns nothing and coverage.py treats that as fatal, aborting the whole report.

This is already known — .coveragerc carries an ignore list with the comment "Omits specific definition files that causes plugin errors". The problem is that it enumerates three files by hand:

omit =
    */windll.pxd
    */_lib/windll.pxd
    */_lib/utils.pxd

Two new files of that kind have since landed and were never added:

Nothing warns you when a new .pxd/.pxi lands. Every one is a latent break.

Why it passed once (Aug 23): whether a given file reaches the report depends on whether any of its traced lines actually executed that night, and which offender trips first depends on iteration order. This is a class of failure, not a single regression — which is why the timeline looks flaky and there is no single "breaking commit".

Fix

Replace the enumeration with a pattern, in [run]:

[run]
plugins = Cython.Coverage
core = ctrace
branch = False
relative_files = True
# Cython's plugin cannot build a FileReporter for declaration/include files.
# Pattern, not an enumeration: every new .pxd/.pxi otherwise breaks `coverage html`.
omit =
    */*.pxd
    */*.pxi

It must be [run], not [report]. Verified against coverage 7.15.4:

  • report_core.get_analysis_to_report() calls coverage._get_file_reporters(morfs) before applying the report_omit matcher, and the PluginError is raised inside that call (control.py:1051). A [report] omit entry is evaluated too late to suppress it.
  • The two sections are independent — report_omit does not fall back to run_omit — so moving the list would silently drop the protection that currently works for windll.pxd/utils.pxd.
  • The existing three entries work precisely because they are in [run]: they keep the files out of the measured set at collection time, and both collection jobs pass --cov-config=$REPO_ROOT/.coveragerc (coverage.yml:183-195, :430-449), so the data arrives at the combining runner already filtered.

Glob behaviour confirmed against coverage.files.GlobMatcher: */*.pxd and */*.pxi match both offenders and do not match .pyx or .py files.

Also worth doing

  • Split the mega-step. coverage combine, coverage html, coverage xml and coverage report should be separate steps so the failing command is named in the UI.
  • Consider coverage html -i as belt-and-braces — but only after the omit fix. On its own it would have hidden this for months.

Severity

Not release-blocking; no product code is affected and no test signal is lost. But coverage reporting has been dark since Aug 14, so any coverage regression in that window is invisible.

Metadata

Metadata

Assignees

Labels

CI/CDCI/CD infrastructureP1Medium priority - Should dobugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions