feat(report): leave out, name and fail on the mutants the command cannot compile - #19
Merged
Merged
Conversation
internal/gobuildrunner resolved `go` to an absolute path and internal/commandscope is about to need the same answer, and the two must agree: a scope resolved by one toolchain and a build made by another would compare two compilers' answers and the disagreement would look like a property of the module under test. The resolution moves to internal/gotoolchain, unchanged, with the reasoning that earned it -- PATH decides which compiler runs unless GOROOT is preferred (go:S4036, CWE-426) -- and a test that pins both branches. Its mutants move with it, so mutantsPerReleaseOnThisRepository is unchanged at 873.
Ditto scopes mutants to a change and judges them with one configured command, and those are two different questions: a staged change spanning five packages, judged by a command naming one of them, produces mutants nothing it runs can kill. They survive, and the score mixes 'your tests missed this' with 'your command cannot see this'. internal/commandscope answers the second question without parsing the command. `go test -json` names every package it executes -- including the ones with no test files, which it reports as a skip -- and `go list -deps -test` names every package those compile in. The closure is the load-bearing half: a mutant in a dependency of a named package IS killable by that package's tests, so a set comparison against the names alone would accuse a correct run. -test is what makes a test-only import count. A command that emits no stream -- make, gotestsum, a wrapper -- is not guessed at: unknown refuses nothing, because a missing accusation costs a number somebody chose an opaque command to get and a wrong one fails a run that was correct. One process per scope, measured at 0.28-0.31s over this repository's 64 executed packages, and nothing at all for a command ditto cannot read. Reading it needs the baseline's own output, and result.Output answered only for an Ok: a green suite is an Err, and a green suite is exactly the run whose stream carries the scope. Widened to the command's output whichever way it ended.
The laboratory is the only place that can learn what the configured test command executes without paying for it: the baseline run it already makes once per release prints the `go test -json` stream that names every package the command executes, and the sandbox it ran in is the tree the rest is resolved against. So Executes(repository, path) answers whether the command compiles the package that owns a file, and the first call is what pays for the baseline on a release that never asks otherwise. There is no second cost: the same sandbox, the same sync.Once. Held by count, not by argument — one baseline, one toolchain query, however many files ask. It answers true when the question cannot be answered: a command that is not `go test -json` has no readable package scope, and neither does one whose toolchain cannot be asked. A missing accusation leaves a number somebody configured an opaque command to get; a wrong one fails a run that was correct.
…not compile A mutant the test command never compiles is not badly tested, it is unmeasured: nothing the command runs can kill it, so it survives, and a score counting it mixes 'your tests missed this' with 'your command cannot see this'. Measured on the report that asked for this: 43 mutants, 23 killed, 20 survived, 17 of those survivors in one package the command never builds, printed as 0.53 against a bar of 0.80 over a run whose ceiling was 0.605. docs/reports/ditto-mutation-scope.md. So the report separates the class the field already separates, exactly as it does for a mutant that never compiled (Zhu/Hall/May, S = D/(M - E)): out of the numerator and the denominator, named with the packages that hold them, and the run does not pass whatever the ratio over the rest says. The test command is not started for them either -- a guaranteed survivor bought with a full suite run is not evidence about anybody's tests. | Total: 3 Killed: 2 Survived: 1 Unmeasured: 1 | 1 of the 4 mutants in this scope are never compiled by this test command, so | nothing it runs can kill them and they are out of the score entirely: | internal/untested (1) exit 3 The exit code is the half a wrapper needs: a scope ditto cannot measure and a score below the bar were both 1, and the responses differ -- the second is answered by testing more, the first by naming every package the scope mutates in --test-command or by narrowing the scope, and by no test at all. Two decorators forward the new count, the test double applies the same exclusion, reporter's summary is one pass instead of three, and --test-command's help now names the lever that exists: it said 'name the package that owns the change', which is true only while the scope holds one. The golden moved by one line and it is the only line it moved: asking the laboratory which packages the command can execute is what pays for the baseline, so the baseline announcement now prints before the first file's announcement rather than between it and the first mutant. No verdict moved and every address is identical.
… cannot compile part of its scope The fixture holds all three classes at once, which is what makes it worth its runtime: two mutants the named package's tests kill, one that survives them, and one in a package the command never compiles. The middle package is the load-bearing one -- `shared` is not named either, and it is NOT unmeasured, because the named package's tests import it and its mutant dies to them. A check that compared names instead of reading the toolchain's closure fails this test. The exit code is pinned beside the output, because it is the half a wrapper reads, and the control is the same fixture under a command that names every package: nothing unmeasured, exit 0, and the shared mutant still judged. So the failure the first half asserts is about the command's scope and not the fixture. Generated with DITTO_GOLDEN_UPDATE=1 and read before it was trusted.
The report asked for it in the same breath as the unmeasurable-score fix, and the fix is what makes it necessary: a run now fails when the test command cannot compile part of the scope, and the honest answer is usually to narrow the run to the package the command names rather than to widen the command. One flag does that; --exclude-prefix needs one flag per other package the change happens to touch. The public shape changes with it, and that is the point rather than a side effect: PlanStaged, RunStaged, PlanChanged and RunChanged take a ditto.Prefixes instead of one []string, so choosing what a run is about and narrowing it are one argument instead of two that can drift apart. An empty entry in either list is ignored rather than honoured, on both sides and for the same reason: every path starts with the empty string, so a list carrying one empty entry means 'no filter' when it was built and 'nothing at all' when it was not -- and the second reading silently mutates no files. Include narrows and exclude still removes, which the tests hold as a pair: a file has to survive both to be planned. -h names the pairing, because the moment a reader needs it is the moment a run just refused their scope.
… not The readme gains the section the report asked for, with the real box: what a run prints when its command cannot compile part of the scope, why each of the three decisions is there -- out of both sides, not run, exit 3 -- and the two limits where ditto says nothing rather than guessing. docs/metrics.md gains the classification row, because that table is where this repository decides what a verdict means, and this is the first row whose fix is not in the tests. docs/backlog.md gains entries 28 and 29: the two ways the check stays silent, and per-package runs as a cost-model change with the measurement that would have to argue for it. The learning log gains four lines, three of which are things that cost a measurement to find: a name comparison would have failed correct runs, result.Output answered only for a kill, and asking a question that pays for the baseline moves a line in the report. docs/reports/ditto-mutation-scope.md is committed with the release it produced, in the reporting repository's words rather than paraphrased.
.golangci.yml sets fix: true, so the pre-commit gate writes wsl's whitespace fix into the working tree on every run — the file is never committed dirty, but it stays dirty after every commit until the fix is carried. One line, and this is the commit that carries it rather than the next one rediscovering it.
The MUTATE pass over this branch removed the nil check Executes made on its scope and every test still passed, which is what that check was worth: verifyBaseline always produces a scope — commandscope.New answers for an empty stream by declining to refuse anything — so the branch was unreachable, and a guard whose removal is invisible is dead weight wearing defense as a costume. The fail-open it looked like it was doing lives one layer down, where a test holds it: the scope answers true when the command named no package. Deleting this one is what makes that the only place the rule lives. mutantsPerReleaseOnThisRepository 949 -> 948, and the ratchet asked for the deletion to be recorded rather than kept as an unclaimed gain.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Answers
docs/reports/ditto-mutation-scope.md, committed here in the reportingrepository's words.
The defect.
ditto stagedscoped 43 mutants to a staged change spanning fivepackages and judged them with one command naming one of them. 17 of the 20
survivors lived in a package that command never compiles, so nothing it ran could
have killed them: the run's ceiling was 0.605, and the number printed as a verdict
was 0.53 against a bar of 0.80. The score was measuring the tests and the command's
scope together, and nothing in the output told the two apart.
The fix. A release now reads which packages the test command executes — from
the command's own
go test -jsonstream, which the baseline run already produces,resolved through
go list -deps -testso a mutant in a dependency of a namedpackage is correctly judged rather than accused. The mutants it cannot compile:
compiled does (
S = D/(M − E)), so the printed score is a real measurement ofwhat could be judged;
not evidence about anybody's tests;
can tell "unmeasurable scope" from "below the bar" — both were 1 before, and the
responses differ: no test answers the first.
--include-prefixarrives with it, the mirror of--exclude-prefix, because thefix makes it necessary: narrowing the run to the package the command names is the
honest pass the report wanted, in one flag instead of one exclusion per package the
change touches.
Evidence, not claims.
testdata/unmeasuredprojectholds all three classes atonce — a package the command names, one its tests import (not unmeasured: its mutant
dies to the named package's tests), and one nothing compiles. Its golden pins the
output, the exit code, and a control where the same fixture under
./...reportsnothing unmeasured and exits 0. The same fixture was also run through the built
binary in a throwaway git repository, which is where the box and the exit code below
were read from.
Two limits, stated rather than hidden (
docs/backlog.mdentry 28): a--test-commandthat emits no stream —make,gotestsum, a wrapper — gets nocheck at all, and the check is per package rather than per file, so a build-tagged
file for another OS stays an ordinary survivor.
Breaking
not run them; the score printed is over the mutants it could judge.
PlanStaged,RunStaged,PlanChangedandRunChangedtake aditto.Prefixesinstead of one
[]string(one-line change per call site).before the first file's announcement, because asking the laboratory which packages
the command compiles is what pays for the baseline. No verdict moved and every
mutant address is identical.
Verified locally
gofmt -l .clean,golangci-lint0 issues, the full suite green with a cold cache(
go test -count=1 ./..., 85s, including the new golden), andmutantsPerReleaseOnThisRepositoryratcheted 873 → 949 with per-file attribution inperf/baseline.json. Not run here: the-raceleg — this shell's C compilerpath is broken (
cgo: C compiler "C:\Program" not found), so CI's devbox leg is theone that runs it.
Please rebase rather than squash: the branch is eight work units, each with its
own tests and its reason, and a squash would collapse them into one entry.