Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
5d58801
test(perfbench): measure what one module-scope build replaces
Disble Sep 18, 2026
3d35fd5
feat(gobuildrunner): run the complete module scope from one compilation
Disble Sep 18, 2026
8a3c6eb
feat(gated): replace only the configured scope, and say when it does not
Disble Sep 18, 2026
360f794
docs: keep an append-only record of the core-performance request
Disble Sep 18, 2026
243f2ad
docs(performance): record the work units this change landed as
Disble Sep 18, 2026
1ef57d5
fix(gobuildrunner): carry the verdict reason onto the module path
Disble Sep 18, 2026
a445391
docs(experiments): measure the gated path through the shipped binary
Disble Sep 18, 2026
8b72c9d
test(perfbench): measure the reason a module-path kill carries
Disble Sep 18, 2026
bb5811e
docs(experiments): repeat the end-to-end measurement across rotated r…
Disble Sep 18, 2026
2deb438
docs(performance): record that the module path scales with the packag…
Disble Sep 18, 2026
913dd4f
test(perfbench): measure the observability closure against the full p…
Disble Sep 18, 2026
cd24b12
docs(performance): record the observability closure result
Disble Sep 18, 2026
3955f33
feat(gobuildrunner): run only the packages that can observe the mutation
Disble Sep 18, 2026
c4c2d25
docs(performance): record the implemented closure and the bent cliff
Disble Sep 18, 2026
5f947d9
docs(performance): confirm the ten-package ratio across rotated rounds
Disble Sep 18, 2026
b89e5dc
test: measure the closure at both ends of a chain, and refute the pre…
Disble Sep 18, 2026
133e040
docs(performance): record the chain refutation rather than explaining…
Disble Sep 18, 2026
374035b
test: check whether the closure engaged at all on the chain fixtures
Disble Sep 18, 2026
828223c
docs(performance): record that the closure engaged and the time still…
Disble Sep 18, 2026
ea13225
test: split the phases, and find that the compile is charged per sour…
Disble Sep 18, 2026
616c842
docs(performance): record the phase split and the per-source-file com…
Disble Sep 18, 2026
c049ab7
test: price the per-batch compile before building anything for it
Disble Sep 18, 2026
4e1c6bd
test: build the shared compilation directory, measure it, and revert it
Disble Sep 18, 2026
ad71822
docs(performance): consolidate the metrics this work moved, and the o…
Disble Sep 18, 2026
7e30910
feat(gated): reuse one sandbox and one compilation directory per release
Disble Sep 18, 2026
e470885
docs(performance): fold the shared-compilation result into the metrics
Disble Sep 18, 2026
85f2ea7
docs(performance): record that the prize was collected once the paths…
Disble Sep 18, 2026
3376a73
feat(gobuildrunner): compile colliding test binaries in separate batches
Disble Sep 19, 2026
5320273
docs(performance): record the collision fix and reconcile two stale c…
Disble Sep 19, 2026
947fe7f
docs(release): prepare 0.11.0
Disble Sep 19, 2026
0d315b2
test(gobuildrunner): cover the Windows case-fold in batch validation
Disble Sep 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,48 @@ All notable changes to this project are documented here. The format follows
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project
adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.11.0] - 2026-09-19

### Changed

- **`Gated()` answers the question the test command asks, not the one the
mutated file's package does.** It used to compile the mutated file's own
package with `go test -c`, which is a narrower question than the default
`go test -count=1 ./...`: a mutant in package P that only a test in a package
importing P can kill survived it, and nothing in the output could see the
difference — the survivor addresses matched, the totals could match, and the
run was faster either way. A gated run now discovers the module's package
layout once, compiles the complete default scope's test binaries, and runs
each of them for every selected mutant.

Gating replaces only the commands whose complete execution plan has been
measured: `go test -count=1 ./...` and its built-in `-json` form in either
flag order. Any custom or unsupported `WithTestCommand` — a package-local
scope, an extra flag, extra spacing, an alternate executable — keeps the
ordinary laboratory and the runner exactly as configured, counted as fallen
back, instead of silently receiving a package-local `go test -c` it never
asked for. The set is closed on purpose: a command that is almost the default
is not the default.

This is a behavior change for anyone who paired `Gated()` with a custom test
command: they now get their own command. That is the point — the old behavior
answered a smaller question than they asked.

It stays opt-in, and no speedup is claimed here that was not measured; the
experiment notes record what the gated path costs and where its gains stop.

### Fixed

- **A module whose test binaries share a basename gates instead of refusing.**
The module-scope runner named every test binary after its package, so on a
tree where two packages produced the same name — a root module, `cmd/x` and
`internal/x` all build `x.test` — it refused before compiling anything,
Gated() fell back for every file, and the repository gated nothing while
looking like any other gated run. Packages are now planned into batches free
of name collisions (case-folded on Windows, where the filesystem is), and
each batch compiles into its own directory. A collision-free module still
plans exactly one batch, so the one-compilation win is unchanged.

## [0.10.0] - 2026-08-30

### Fixed
Expand Down Expand Up @@ -703,6 +745,7 @@ here, not yet built.
- The `retract` block. It named published versions of the upstream module path,
which do not exist under this one.

[0.11.0]: https://github.com/Disble/ditto/releases/tag/v0.11.0
[0.10.0]: https://github.com/Disble/ditto/releases/tag/v0.10.0
[0.9.0]: https://github.com/Disble/ditto/releases/tag/v0.9.0
[0.8.0]: https://github.com/Disble/ditto/releases/tag/v0.8.0
Expand Down
24 changes: 21 additions & 3 deletions cmd/ditto/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ func runCommand(args []string) error {
root := flags.String("root", ".", "repository root to mutate")
testCommand := flags.String("test-command", "go test -count=1 -json ./...", testCommandHelp)
threshold := flags.Float64("threshold", 1.0, "minimum mutation score, from 0 to 1")
gated := flags.Bool("gated", false, "run a file's mutants from one compilation instead of one each")
gated := flags.Bool("gated", false, gatedHelp)
confirm := flags.Bool("confirm-kills", false, confirmKillsHelp)
loud := flags.Bool("verbose", false, "print what the run is doing as it does it")
sandbox := flags.String("sandbox", "", `how each file reaches the sandbox: "copy" (default), "hardlink" or "link"`)
Expand Down Expand Up @@ -204,7 +204,7 @@ func stagedCommand(args []string, out io.Writer) error {
testCommand := flags.String("test-command", "go test -count=1 -json ./...", testCommandHelp)
threshold := flags.Float64("threshold", 1.0, "minimum mutation score, from 0 to 1")
dry := flags.Bool("dry", false, "report what the staged change justifies and run nothing")
gated := flags.Bool("gated", false, "run a file's mutants from one compilation instead of one each")
gated := flags.Bool("gated", false, gatedHelp)
confirm := flags.Bool("confirm-kills", false, confirmKillsHelp)
loud := flags.Bool("verbose", false, "print what the run is doing as it does it")
sandbox := flags.String("sandbox", "", `how each file reaches the sandbox: "copy" (default), "hardlink" or "link"`)
Expand Down Expand Up @@ -288,7 +288,7 @@ func changedCommand(args []string, out io.Writer) error {
testCommand := flags.String("test-command", "go test -count=1 -json ./...", testCommandHelp)
threshold := flags.Float64("threshold", 1.0, "minimum mutation score, from 0 to 1")
dry := flags.Bool("dry", false, "report what the change justifies and run nothing")
gated := flags.Bool("gated", false, "run a file's mutants from one compilation instead of one each")
gated := flags.Bool("gated", false, gatedHelp)
confirm := flags.Bool("confirm-kills", false, confirmKillsHelp)
loud := flags.Bool("verbose", false, "print what the run is doing as it does it")
sandbox := flags.String("sandbox", "", `how each file reaches the sandbox: "copy" (default), "hardlink" or "link"`)
Expand Down Expand Up @@ -433,6 +433,24 @@ const testCommandHelp = "the `command` that decides whether a mutant died. It ru
"that owns the change instead. -json is what lets ditto say WHY a mutant died; without it a " +
"mutant that never compiled is counted as killed"

// gatedHelp is the description of --gated on all three subcommands, one
// constant so the three cannot drift apart.
//
// The behavior it describes changed in 0.11.0 and the old line described the
// behavior that went away: Gated() no longer compiles "a file's" package -- it
// compiles the complete module scope, and only when the configured command is
// the exact default, in either -json order. A custom command keeps its own
// ordinary path, which is the part a reader pairing --gated with
// --test-command most needs to know at the moment of typing.
//
// No backquoted word anywhere in this string: flag.PrintDefaults takes the
// first one as the flag's VALUE NAME, and --gated is a bool flag that must
// never render as taking an argument.
const gatedHelp = "run the module's mutants from one compilation instead of one test-command start " +
"each (750-950 ms per start). Only the exact default command go test -count=1 ./... -- with or " +
"without -json -- is gated; any custom --test-command keeps its ordinary path, and mutants that " +
"cannot be gated keep their own"

// confirmKillsHelp names the cost as well as the behaviour, for the reason
// testCommandHelp does.
const confirmKillsHelp = "re-run a mutant that died by assertion, once, and believe the second answer " +
Expand Down
43 changes: 43 additions & 0 deletions cmd/ditto/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,49 @@ func TestTestCommandHelpRendersItsValueName(t *testing.T) {
assert.NotContains(t, rendered.String(), "-test-command ./...")
}

// TestGatedHelpCopy pins what --gated's -h line promises, because the behavior
// behind the flag changed in 0.11.0: Gated() now answers the complete module
// scope and only the exact default commands, so the previous wording -- "run a
// file's mutants from one compilation" -- described a package-local build that
// no longer exists, on all three subcommands at once.
func TestGatedHelpCopy(t *testing.T) {
t.Run("says the scope is the whole module", func(t *testing.T) {
assert.Contains(t, gatedHelp, "module")
})

t.Run("names the exact commands that are eligible", func(t *testing.T) {
assert.Contains(t, gatedHelp, "go test -count=1 ./...")
assert.Contains(t, gatedHelp, "-json")
})

t.Run("says a custom test command keeps its own path", func(t *testing.T) {
assert.Contains(t, gatedHelp, "custom")
})

// The old line promised one compilation for "a file's" mutants, which was
// the package-scope behavior. Nothing in the help may claim it again.
t.Run("does not name the mutated file as the compilation unit", func(t *testing.T) {
assert.NotContains(t, gatedHelp, "a file's mutants")
})
}

// TestGatedHelpRendersWithoutAValueName is the same guard
// TestTestCommandHelpRendersItsValueName holds: the first backquoted word in a
// usage string becomes the flag's VALUE NAME, and --gated is a bool flag that
// must never render as though it takes one. A backtick anywhere in gatedHelp
// would silently turn `-gated` into `-gated something`.
func TestGatedHelpRendersWithoutAValueName(t *testing.T) {
rendered := &bytes.Buffer{}

flags := flag.NewFlagSet("ditto run", flag.ContinueOnError)
flags.SetOutput(rendered)
flags.Bool("gated", false, gatedHelp)
flags.PrintDefaults()

assert.Contains(t, rendered.String(), "-gated")
assert.NotContains(t, rendered.String(), "-gated ", "a value name rendered after the bool flag")
}

// TestChangedRefusesToGuessABase covers the one decision `changed` deliberately
// does not make for you. There is no default that is right on a CI checkout, in
// a working tree and on a branch at once, and a base guessed wrong is either a
Expand Down
Loading
Loading