Export coverage as Cobertura XML

CHANGELOG.md

@@ -22,6 +22,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - The JSON-RPC protocol gained a client → controller `shutdown` notification for a graceful shutdown: every run is cancelled, every test process is terminated (force-killed if it does not exit within the grace period) and the controller exits once they are gone.
 - `write_junit_xml(io_or_path, ::TestrunResult; root)` writes a test run as JUnit XML, the one report format every CI system ingests. Test items are grouped into one `<testsuite>` per source file and one `<testcase>` per (item × run profile); captured output goes to `<system-out>` with ANSI escape sequences stripped, and performance statistics become `<properties>`. It is a pure function of a `TestrunResult`, so a result file written by one process can be converted by another.
 - `write_lcov(io_or_path, ::TestrunResult; root)` writes a run's merged coverage in LCOV info format, so consumers no longer have to reach into the vendored `CoverageTools`. `root` is a directory path or `file:` URI that the `SF:` paths are relativized against, mirroring `write_junit_xml` — coverage services match `SF:` paths against paths in the repository, and the absolute paths of a CI runner match nothing at all, which is one way a fully covered package comes out at 0%. A file outside `root` keeps its absolute path rather than a `..`-heavy one, the same choice the JUnit writer makes, and paths always use `/` separators so a Windows leg and a Linux leg of the same matrix contribute the same file names to a merged report.
+- `write_cobertura(io_or_path, ::TestrunResult; root)` writes a run's merged coverage as a Cobertura XML report — the sibling of `write_lcov`, with the same `root` relativization, the same `/` separators and the same "returns `false` and writes nothing when the run collected no coverage" contract, so a caller can drive both from one `if`. It exists because GitHub Code Quality's coverage feature takes Cobertura and nothing else. Cobertura describes branch and method coverage too; Julia's line-count instrumentation produces neither, so those attributes are present and always zero, and a line Julia could not instrument is left out rather than reported as a line nobody ran.
 - `TestrunResultTestitemProfile` gained a `perf` field and `TestrunResult` a `coverage` field. Both are optional and both are read tolerantly: a result file written before these existed still parses, which matters because `julia-report-ci-results` merges files produced by every leg of a CI matrix and those legs are not necessarily on the same version.
 - `TestItemDetail` gained `option_skip`, carrying the `@testitem` `skip` kwarg — either a literal `Bool` or the source text of an expression, which is evaluated in the test process rather than the controller so it sees the test process's Julia version and platform.
 - The terminal test item callbacks take an optional trailing argument: performance statistics for `passed`/`failed`/`errored`, and a reason for `skipped`. Callbacks written against the previous signatures are unaffected — the controller falls back to the arity the callback accepts.

Project.toml

@@ -1,7 +1,7 @@
 name = "TestItemControllers"
 uuid = "340407ab-12a4-4a78-8da2-3de3b3fd1448"
 authors = ["David Anthoff <anthoff@berkeley.edu>"]
-version = "1.11.3-DEV"
+version = "1.12.0-DEV"
 
 [deps]
 Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"

src/TestItemControllers.jl

@@ -27,7 +27,7 @@ export execute_testrun
 export TestrunResult, TestrunResultTestitem, TestrunResultTestitemProfile,
     TestrunResultMessage, TestrunResultStackFrame, TestrunResultDefinitionError,
     TestrunResultPerfStats, TestrunResultFileCoverage
-export write_junit_xml, write_lcov
+export write_junit_xml, write_lcov, write_cobertura
 
 include("json_protocol.jl")
 include("../shared/testserver_protocol.jl")
@@ -40,6 +40,7 @@ include("results.jl")
 using .Results
 include("junit.jl")
 include("lcov.jl")
+include("cobertura.jl")
 include("testenvironment.jl")
 
 include("fsm.jl")

src/cobertura.jl

@@ -0,0 +1,170 @@
+"""
+Cobertura XML export for the coverage carried on a [`TestrunResult`](@ref).
+
+The sibling of [`write_lcov`](@ref), for the consumers that take Cobertura rather than
+LCOV — GitHub Code Quality's coverage feature takes Cobertura and nothing else.
+
+Cobertura describes branch and method coverage too. Julia's line-count instrumentation
+produces neither, and the one consumer this exists for reads line coverage only, so the
+branch and complexity attributes are present (the format's consumers expect them) and
+always zero.
+"""
+
+"""
+    write_cobertura(io_or_path, result::TestrunResult; root=nothing)
+
+Write the merged coverage of `result` as a Cobertura XML report.
+
+Returns `false` and writes nothing when the run collected no coverage — that is the normal
+outcome for a run that was not started in coverage mode, not an error. The same contract as
+[`write_lcov`](@ref), so a caller can drive both from the same `if`.
+
+`root` is a directory path or `file:` URI to relativize the `filename` attributes against;
+without it they are absolute. Coverage services match those paths against paths in the
+repository, and the absolute paths of a CI runner match nothing at all — which is one way a
+fully covered package comes out at 0%. A file outside `root` keeps its absolute path rather
+than a `..`-heavy one, the same choice [`write_junit_xml`](@ref) makes.
+
+Paths always use `/` separators, so a Windows leg and a Linux leg of the same matrix
+contribute the same file names to a merged report.
+
+Entries whose URI is not a `file:` URI are skipped.
+"""
+function write_cobertura(io::IO, result::TestrunResult; root::Union{Nothing,AbstractString}=nothing)
+    files = _cobertura_files(result, root)
+    files === nothing && return false
+    _write_cobertura(io, files)
+    return true
+end
+
+function write_cobertura(path::AbstractString, result::TestrunResult; root::Union{Nothing,AbstractString}=nothing)
+    files = _cobertura_files(result, root)
+    files === nothing && return false
+    open(path, "w") do io
+        _write_cobertura(io, files)
+    end
+    return true
+end
+
+# `(filename, lines)` per source file, where `lines` is the `(number, hits)` of every
+# instrumentable line. A line Julia could not instrument has no representation in Cobertura
+# — there is no "not applicable" hit count — so it is left out rather than reported as a
+# line nobody ran.
+function _cobertura_files(result::TestrunResult, root::Union{Nothing,AbstractString})
+    result.coverage === nothing && return nothing
+    isempty(result.coverage) && return nothing
+
+    files = Tuple{String,Vector{Tuple{Int,Int}}}[]
+    for fc in result.coverage
+        # Shared with the JUnit and LCOV writers: same relativization, same `/` separators,
+        # same refusal to walk out of the root with `..`.
+        filename = _report_path(fc.uri, root)
+        filename === nothing && continue
+        lines = Tuple{Int,Int}[]
+        for (i, hits) in enumerate(fc.coverage)
+            hits === nothing && continue
+            push!(lines, (i, hits))
+        end
+        push!(files, (filename, lines))
+    end
+
+    return isempty(files) ? nothing : files
+end
+
+# Cobertura's `line-rate` is covered lines over instrumentable lines. A file with nothing to
+# instrument is not 0% covered, it is undefined; 1.0 is what every producer emits for it and
+# what keeps it from dragging a project's rate down.
+function _cobertura_rate(covered::Integer, valid::Integer)
+    valid == 0 && return 1.0
+    return covered / valid
+end
+
+_cobertura_counts(lines) = (count(l -> l[2] > 0, lines), length(lines))
+
+# Cobertura groups classes into packages. Nothing reads the grouping for a language without
+# packages, but a report that puts every file in one anonymous bucket is unreadable in the
+# viewers that do show it, so the directory is used — dotted, as the format's Java ancestry
+# expects.
+function _cobertura_package_name(filename::AbstractString)
+    dir = dirname(filename)
+    isempty(dir) && return "."
+    return replace(dir, '/' => '.')
+end
+
+function _write_cobertura(io::IO, files::Vector{Tuple{String,Vector{Tuple{Int,Int}}}})
+    total_covered = 0
+    total_valid = 0
+    for (_, lines) in files
+        covered, valid = _cobertura_counts(lines)
+        total_covered += covered
+        total_valid += valid
+    end
+
+    # Cobertura timestamps are milliseconds since the epoch. `Int64` explicitly, not
+    # `Int`: on a 32-bit Julia that is `Int32`, and epoch milliseconds passed what an
+    # `Int32` can hold in 1970.
+    timestamp = round(Int64, time() * 1000)
+
+    println(io, "<?xml version=\"1.0\" encoding=\"UTF-8\"?>")
+    println(io, "<!DOCTYPE coverage SYSTEM \"http://cobertura.sourceforge.net/xml/coverage-04.dtd\">")
+    println(io, "<coverage line-rate=\"", _cobertura_rate(total_covered, total_valid),
+        "\" branch-rate=\"0.0\" lines-covered=\"", total_covered,
+        "\" lines-valid=\"", total_valid,
+        "\" branches-covered=\"0\" branches-valid=\"0\" complexity=\"0\" version=\"2.0.3\" timestamp=\"",
+        timestamp, "\">")
+    # The filenames are already relative to the root the caller gave us, so the source root
+    # a consumer should join them against is wherever it checked the repository out.
+    println(io, "  <sources>")
+    println(io, "    <source>.</source>")
+    println(io, "  </sources>")
+    println(io, "  <packages>")
+
+    for (package, members) in _group_by_package(files)
+        pkg_covered = 0
+        pkg_valid = 0
+        for (_, lines) in members
+            covered, valid = _cobertura_counts(lines)
+            pkg_covered += covered
+            pkg_valid += valid
+        end
+        println(io, "    <package name=\"", _xml_escape(package), "\" line-rate=\"",
+            _cobertura_rate(pkg_covered, pkg_valid), "\" branch-rate=\"0.0\" complexity=\"0\">")
+        println(io, "      <classes>")
+        for (filename, lines) in members
+            covered, valid = _cobertura_counts(lines)
+            println(io, "        <class name=\"", _xml_escape(basename(filename)),
+                "\" filename=\"", _xml_escape(filename), "\" line-rate=\"",
+                _cobertura_rate(covered, valid), "\" branch-rate=\"0.0\" complexity=\"0\">")
+            println(io, "          <methods/>")
+            println(io, "          <lines>")
+            for (number, hits) in lines
+                println(io, "            <line number=\"", number, "\" hits=\"", hits, "\" branch=\"false\"/>")
+            end
+            println(io, "          </lines>")
+            println(io, "        </class>")
+        end
+        println(io, "      </classes>")
+        println(io, "    </package>")
+    end
+
+    println(io, "  </packages>")
+    println(io, "</coverage>")
+
+    return nothing
+end
+
+# Package order follows first appearance, and files keep the order the result gave them, so
+# the same run always produces the same bytes.
+function _group_by_package(files::Vector{Tuple{String,Vector{Tuple{Int,Int}}}})
+    order = String[]
+    groups = Dict{String,Vector{Tuple{String,Vector{Tuple{Int,Int}}}}}()
+    for entry in files
+        package = _cobertura_package_name(entry[1])
+        members = get!(groups, package) do
+            push!(order, package)
+            Tuple{String,Vector{Tuple{Int,Int}}}[]
+        end
+        push!(members, entry)
+    end
+    return [(package, groups[package]) for package in order]
+end

test/test_cobertura.jl

@@ -0,0 +1,141 @@
+@testitem "Cobertura export" begin
+    using TestItemControllers: write_cobertura
+    using TestItemControllers.Results
+
+    empty_result = TestrunResult(TestrunResultDefinitionError[], TestrunResultTestitem[], Dict{String,String}())
+
+    io = IOBuffer()
+    @test write_cobertura(io, empty_result) == false
+    @test isempty(take!(io))
+
+    covered = TestrunResult(
+        TestrunResultDefinitionError[],
+        TestrunResultTestitem[],
+        Dict{String,String}(),
+        [TestrunResultFileCoverage("file:///c%3A/pkg/src/f.jl", Union{Nothing,Int}[nothing, 3, 0, nothing])],
+    )
+
+    io = IOBuffer()
+    @test write_cobertura(io, covered; root="file:///c%3A/pkg") == true
+    xml = String(take!(io))
+
+    @test startswith(xml, "<?xml version=\"1.0\" encoding=\"UTF-8\"?>")
+    @test occursin("<coverage ", xml)
+    @test occursin("</coverage>", xml)
+    @test occursin("filename=\"src/f.jl\"", xml)
+    @test occursin("name=\"f.jl\"", xml)
+    @test occursin("<package name=\"src\"", xml)
+    # Two instrumentable lines, one of them run.
+    @test occursin("lines-valid=\"2\"", xml)
+    @test occursin("lines-covered=\"1\"", xml)
+    @test occursin("line-rate=\"0.5\"", xml)
+    @test occursin("<line number=\"2\" hits=\"3\" branch=\"false\"/>", xml)
+    @test occursin("<line number=\"3\" hits=\"0\" branch=\"false\"/>", xml)
+    # A line Julia could not instrument has no representation in Cobertura, so it is left
+    # out rather than reported as a line nobody ran.
+    @test !occursin("number=\"1\"", xml)
+    @test !occursin("number=\"4\"", xml)
+
+    # Epoch milliseconds overflow an `Int32`, so on a 32-bit Julia a `round(Int, ...)`
+    # here throws an InexactError and no report gets written at all.
+    stamp = match(r"timestamp=\"(\d+)\"", xml)
+    @test stamp !== nothing
+    @test parse(Int64, stamp[1]) > typemax(Int32)
+end
+
+@testitem "Cobertura export escapes and skips what it cannot report" begin
+    using TestItemControllers: write_cobertura
+    using TestItemControllers.Results
+
+    # The consumers parse this with a real XML parser, so an unescaped character in a path
+    # is a silently rejected report rather than a visibly broken one.
+    result = TestrunResult(
+        TestrunResultDefinitionError[],
+        TestrunResultTestitem[],
+        Dict{String,String}(),
+        [
+            TestrunResultFileCoverage("file:///c%3A/pkg/src/a%20%26%20b.jl", Union{Nothing,Int}[1, 0]),
+            TestrunResultFileCoverage("file:///c%3A/pkg/src/sub/c.jl", Union{Nothing,Int}[nothing]),
+            # Not a `file:` URI, so it has no path to report and is skipped.
+            TestrunResultFileCoverage("untitled:Untitled-1", Union{Nothing,Int}[1]),
+        ],
+    )
+
+    io = IOBuffer()
+    @test write_cobertura(io, result; root="file:///c%3A/pkg") == true
+    xml = String(take!(io))
+
+    @test occursin("filename=\"src/a &amp; b.jl\"", xml)
+    @test !occursin("Untitled-1", xml)
+    # A directory becomes a dotted package name, the way the format's Java ancestry expects.
+    # A file with nothing to instrument is not 0% covered, it is undefined -- reporting it
+    # as zero would drag the project's rate down for a file with no code in it.
+    @test occursin("<package name=\"src.sub\" line-rate=\"1.0\"", xml)
+    # The whole run: one covered line out of two instrumentable ones, the empty file adding
+    # nothing to either count.
+    @test occursin("lines-valid=\"2\"", xml)
+    @test occursin("lines-covered=\"1\"", xml)
+end
+
+@testitem "Cobertura export relativizes against a root" begin
+    using TestItemControllers: write_cobertura, filepath2uri
+    using TestItemControllers.Results
+
+    # Coverage services match the `filename` attributes against paths in the repository.
+    # The absolute paths of a CI runner match nothing at all, which is one way a fully
+    # covered package gets reported as 0%.
+    mktempdir() do dir
+        dir = realpath(dir)
+        pkg = joinpath(dir, "pkg")
+        mkpath(joinpath(pkg, "src"))
+        uri = string(filepath2uri(joinpath(pkg, "src", "f.jl")))
+
+        result = TestrunResult(
+            TestrunResultDefinitionError[],
+            TestrunResultTestitem[],
+            Dict{String,String}(),
+            [TestrunResultFileCoverage(uri, Union{Nothing,Int}[nothing, 3, 0])],
+        )
+
+        io = IOBuffer()
+        @test write_cobertura(io, result; root=pkg) == true
+        xml = String(take!(io))
+
+        @test occursin("filename=\"src/f.jl\"", xml)
+        @test !occursin(replace(pkg, "\\" => "/"), xml)
+
+        # ...and a relative root is resolved against the working directory, which `relpath`
+        # does not do on its own.
+        io = IOBuffer()
+        cd(dir) do
+            write_cobertura(io, result; root="pkg")
+        end
+        @test occursin("filename=\"src/f.jl\"", String(take!(io)))
+    end
+end
+
+@testitem "Cobertura export writes to a path" begin
+    using TestItemControllers: write_cobertura
+    using TestItemControllers.Results
+
+    empty_result = TestrunResult(TestrunResultDefinitionError[], TestrunResultTestitem[], Dict{String,String}())
+
+    covered = TestrunResult(
+        TestrunResultDefinitionError[],
+        TestrunResultTestitem[],
+        Dict{String,String}(),
+        [TestrunResultFileCoverage("file:///c%3A/pkg/src/f.jl", Union{Nothing,Int}[1])],
+    )
+
+    mktempdir() do dir
+        path = joinpath(dir, "cobertura.xml")
+
+        # A run without coverage must not leave an empty file behind: an empty report is
+        # indistinguishable from a real 0% one to whatever uploads it.
+        @test write_cobertura(path, empty_result) == false
+        @test !isfile(path)
+
+        @test write_cobertura(path, covered; root="file:///c%3A/pkg") == true
+        @test occursin("filename=\"src/f.jl\"", read(path, String))
+    end
+end