feat(gate): mutate the change, so the gate finishes - #12
Merged
Merged
Conversation
Backlog entry 21, closed by building the answer it already named. The repository-sized gate asks for 736 mutants and dies at its thirty minutes having reached about 424, measured four times including once on the release this closes. Both levers are spent: gating removes 54% of the compilations and does not close it, and cutting the mutant's suite by 46% moved the gate by 0.5% because -failfast already stops a killed mutant at its first failing test. The bill is the wrong SIZE rather than badly paid. `ditto staged` could not be the gate, which is why this stayed open after the staged path shipped: it reads the index, and on a CI checkout nothing is staged, so it would skip and report a green that measured nothing -- the exact shape of failure mutation.yml already refuses in its own comments. So the scope had to learn to read a RANGE. PlanChanged and RunChanged ask the staged question of `base...HEAD` instead of `--cached`, reusing the diff parsing, the byte offsets, the fail-open rule and the sandbox unchanged. Three dots rather than two, so a base that has moved on does not drag somebody else's commits into the bill. RunChanged refuses a checkout with uncommitted work in it, and that refusal is the whole safety of reusing the index-backed sandbox: a range scope names bytes of HEAD, and those are the same bytes only while nothing is modified or staged. Scoping against one tree and mutating another is the defect already measured at seven of eight verdicts moving, and it is silent. `ditto changed --since <ref>` exposes it, with --dry like staged has. The base is never guessed: there is no default that is right in CI, in a working tree and on a branch at once, and a scope guessed wrong is either a bill nobody asked for or a green that measured nothing. CI runs `make test.mutation.changed`. `make test.mutation` is untouched and workflow_dispatch still reaches it -- the repository-sized question is worth asking on purpose, just not on every push against a clock it cannot beat. The checkout gains fetch-depth: 0, because a range needs the history it lives in. mutantsPerReleaseOnThisRepository moves 736 to 783. That the change which SHRINKS the gate grows that counter by 47 is the entry rather than a contradiction: the number is what the repository-sized question costs, and the gate has stopped asking it on every push.
The readme gains the CI case the staged section could not answer: a checkout where nothing is staged because the change is already committed. It also says why there is no default base and why a dirty checkout is refused, because both read as fussiness until the reason is on the page. Two tests for the dispatch: `changed` refuses to guess a base, and it appears in the usage a reader gets from `--help` -- a subcommand that was never wired would otherwise look exactly like a typo.
The gate that now finishes reported 47 mutants, 13 killed, 34 survived -- 0.28 against a minimum of 0.50 -- and it was right. The range scope shipped with its error paths untested: every `if err != nil` inverted survived, because nothing had ever made those errors happen. PlanChanged and RunChanged are exercised against real temporary repositories rather than a double. What git says IS the subject here, and a fake that agrees with my reading of git proves my reading rather than the behaviour. Covered: a committed change, a docs-only commit, an excluded prefix, an unknown base, a directory outside any repository, and a dirty checkout. The two fail-open branches are covered by name and not only by their effect. There are two ways to widen a scope here and they mean different things -- a diff nobody could parse, and a range that missed the file -- so the reason is asserted rather than merely its presence. --threshold's bounds are covered on `changed` and, for the first time, on `staged` too, so the two cannot drift apart unnoticed. --dry's report is exercised for both the something and the nothing case, and the dispatch is checked to pass its flags on: one too few drops the first flag, one too many hands `changed` its own name, and both look exactly like a user who forgot --since.
SonarCloud refused the change at 5.2% duplication on new code against a 3% gate, and it was pointing at something real in both places rather than at noise. RunStaged and RunChanged differed in exactly one thing -- which pair of trees the diff is read from -- and shared everything below it: the sandbox built from the index, the .ditto.json for what git does not carry, the widened-scope notice, the generated-path announcements. That is runInSandbox now. Keeping one copy is not tidying: two copies of it drift, and a drift there means one entry point measuring different bytes than the other while both report the same kind of number. The git fixtures were duplicated across two test packages that cannot import each other's helpers, so they move to internal/dittotesting, which is what that package is for. GitRepository, GitRepositoryWithAChange, Git and WriteFile, with the inherited GIT_DIR removed rather than blanked -- git rejects an empty one outright, and an inherited one would point a fixture at the real checkout. mutantsPerReleaseOnThisRepository 783 to 784.
SonarCloud refused the change on a B security rating for new code. `exec.Command("git", …)` reads PATH at every call, so a directory an attacker
can write to -- or prepend -- decides which git runs. That is rule S4036, and
internal/gobuildrunner already resolves the Go toolchain the same way for the
same reason: something ambient deciding what a subprocess really is.
mutantsPerReleaseOnThisRepository 784 to 785.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backlog entry 21, open since it was measured, closed by building the answer it
already named.
v0.8.0 shipped with this repository's own mutation gate red, and I reported that
as pre-existing and moved on. It was pre-existing. It was also the gate, and a
release whose gate does not pass is not a finished release.
The measurements
FAIL … 1800.181s— the-timeout=30mexactly, ~424 of 783 mutants reachedok … 838.835sThe middle row is the point. The gate finished, and what it said was that I had
shipped the range scope with its error paths untested: every
if err != nilinverted survived, because nothing had ever made those errors happen. The last
row is the same gate after the tests that kill them.
Why
ditto stagedcould not be this gateIt reads the index, and a CI checkout has nothing staged — the change is already
committed. Pointed at it, the gate would skip, report success, and measure
nothing: the exact shape of failure
mutation.ymlalready refuses in its owncomments. That is why entry 21 stayed open after the staged path shipped.
What was added
ChangedFilesandChangedScopeOfreadbase...HEADinstead of--cached,reusing the diff parsing, the byte offsets, the fail-open rule and the sandbox
unchanged. Three dots rather than two, so a base that has moved on does not drag
somebody else's commits into the bill.
RunChangedrefuses a checkout with uncommitted work in it, and that refusal isthe whole safety of reusing the index-backed sandbox: a range scope names bytes
of
HEADwhile the sandbox is written from the index, and those are the sametree only while nothing is modified or staged.
ditto changed --since <ref>exposes it, with--dry. There is no default baseand there should not be one: on CI the useful base is the last release, on a
branch it is the trunk, and a base guessed wrong is either a bill nobody asked
for or a scope of nothing reported as green.
What is still open
Eight survivors at 0.83. Four are error paths that need a
.ditto.jsonor afailing
Materialize; one is an equivalent mutant (len(files) == 0→== -1,which cannot change the answer); one is
--threshold's float default; one is arangebreak over the generated-paths loop. They are named here rather thanquietly left at a passing score.
Checks
livetreecounters green, lint0 issues.mutantsPerReleaseOnThisRepository736 → 783. That the change which shrinksthe gate grows that counter by 47 is the entry rather than a contradiction:
the number is what the repository-sized question costs, and the gate has
stopped asking it on every push.