From 1aa0271f3548ae1547e38c2cea54e99c81208df9 Mon Sep 17 00:00:00 2001 From: tucktuck101 Date: Sat, 22 Aug 2026 09:19:10 +1200 Subject: [PATCH 1/2] docs(research): clean-merge-but-wrong -- detection tops out near 32%, and 23 of 27 fork rows are exposed (#363) Signed-off-by: tucktuck101 --- .../363-detecting-clean-but-wrong-merges.md | 188 ++++++++++++++++++ 1 file changed, 188 insertions(+) create mode 100644 launchpad/Research/363-detecting-clean-but-wrong-merges.md diff --git a/launchpad/Research/363-detecting-clean-but-wrong-merges.md b/launchpad/Research/363-detecting-clean-but-wrong-merges.md new file mode 100644 index 00000000000..327aab9010a --- /dev/null +++ b/launchpad/Research/363-detecting-clean-but-wrong-merges.md @@ -0,0 +1,188 @@ +# Detecting a clean merge that is wrong, and how large that class is here + +**Title:** The clean-merge-but-broken class — detection practice, and its size in this fork +**Summary:** The class is real, well-studied and **not automatically solvable**: the best published tool detects 32% of semantic conflicts, and its authors conclude "developers cannot rely solely on such semantic merge tools". Sizes the class here: **23 of the fork's 27 contested files are in-place edits and therefore coupled to upstream structure by construction**; 4 are not. One instance is demonstrated (#360's `relay_url`), one alleged instance is refuted (`bin/.lefthookrc`), and an anchor-survival test across 16 dependencies finds all currently intact — so no *additional* clean break is pending in this drop. Proposes the two cheap mechanical pre-flights that would have caught what was findable. +**Tags:** `upstream-sync` `vendor-drop` `semantic-conflict` `adr-0022` `prd-273` `detection` +**Established:** 2026-08-22 · **Answers:** [#363](https://github.com/launchpad-26/buzz/issues/363) · **Parent:** [#273](https://github.com/launchpad-26/buzz/issues/273) + +**References are pinned.** Fork-side claims cite `launchpad-26/buzz` at +[`5d76799d6e44f2f76aa7bd78c5343d339af98f63`](https://github.com/launchpad-26/buzz/tree/5d76799d6e44f2f76aa7bd78c5343d339af98f63); upstream-side claims cite `block/buzz` at +[`025425591ed67518a63870316f1473ffd02dd520`](https://github.com/block/buzz/tree/025425591ed67518a63870316f1473ffd02dd520). Paths inside fenced blocks are command +*output* and are left unlinked deliberately. + +--- + +## Finding + +**Nobody has solved this, and the research says plainly that nobody is close.** The honest answer to "how do downstreams detect it" is: partially, with tests, at a recall the literature measures at roughly a third. + +But the fork's specific situation is better than that sounds, for a reason worth stating up front: **most of what makes this class dangerous in the literature is invisible interference between two feature changes. This fork's version is narrower** — an upstream restructuring that invalidates a fork edit — and two of its three shapes are mechanically checkable. + +| Shape | Detectable by | Instance here | +|---|---|---| +| Fork edit references something upstream deleted | **Compiler / build** | **Demonstrated** — #360's `relay_url` | +| Fork edit anchors on structure upstream renamed | **Anchor-survival grep** | None pending — 16 anchors tested, all intact | +| Fork edit still applies but no longer means what it did | Tests, at ~32% recall | Unknown, and unknowable cheaply | + +The first two are cheap pre-flights the fork does not currently run. The third is the genuinely hard residue, and it is smaller than the class as a whole. + +--- + +## How the class is detected: the state of the art + +The research definition, from *Detecting Semantic Conflicts with Unit Tests*: + +> "two contributions (sets of changes) to a base program semantically conflict—that is, interfere in an unplanned way—when the specifications they are individually supposed to satisfy are not jointly satisfied by the program that integrates them." + +and the operational version: + +> "semantic conflicts occur when textual merging succeeds and code compiles, yet the integrated changes exhibit undesired behavioral interference" + +**The measured effectiveness is the important part.** Evaluating SAM — a tool that generates unit tests with four generators (EvoSuite, Differential EvoSuite, Randoop, Randoop Clean) and runs them against Base, Left, Right and Merge — across 85 change pairs from 51 merge scenarios: + +- detected **9 of 28 conflicts** — recall **0.32** +- precision **0.84** (3 false positives in 57 non-conflict cases) +- 19 false negatives, of which manual analysis suggested 13 were theoretically reachable with better test generation + +The authors' own conclusion: + +> "developers cannot rely solely on such semantic merge tools for detecting conflicts" + +**So the answer to "how do downstreams detect this" is: they mostly don't, and the ones who try catch about a third.** Anyone proposing a mechanism for #273 that claims to close this hole is overclaiming. + +Practitioner sources agree from the other direction. Electron, which maintains one of the largest downstream patch sets in existence, states it as a fact of life rather than a solved problem: *"When upstream code changes, patches can break—sometimes without even a patch conflict or a compilation error."* And the literature notes the scale problem for exactly this shape of fork: *"For large upstream repositories like Chromium (of which Microsoft Edge is a downstream divergent fork), searching through thousands of upstream commits is tedious and error-prone."* + +### What actually works, in descending order of cost-effectiveness + +1. **Build the merge.** Catches the whole first shape. #360 found the `relay_url` defect this way, and `merge-tree` — which is what every figure in the #273 thread was derived from — could not have. +2. **Run the downstream's own tests.** This is the literature's technique without the automated generation: tests that encode *the fork's* invariants, not upstream's. The fork has one already in [`scripts/test-ci-changed-paths-filter.sh`](https://github.com/launchpad-26/buzz/blob/5d76799d6e44f2f76aa7bd78c5343d339af98f63/scripts/test-ci-changed-paths-filter.sh) — a contract test for a fork divergence — which is exactly the right pattern. +3. **Assert the position survived.** Cheap, arbitrary, and cannot fail silently (see [#362](https://github.com/launchpad-26/buzz/issues/362)). +4. **Generated-test tooling.** ~32% recall, and the tools in the study are Java-only. Not applicable to a Rust/TypeScript/Dart codebase without substantial work, and not worth it at that recall. +5. **Staged or canary adoption.** Standard in distributions; requires somewhere to stage, which for this fork means the VPS, and is out of scope here. + +--- + +## How large the class is in this fork + +### Coupling is determined by the *form* of the divergence + +A divergence that is **append-only or fenced** has no structural dependency on upstream — upstream can rewrite everything around it. An **in-place edit** depends by construction on the surrounding structure still being there and still meaning the same thing. + +Classifying the 27 contested files by form: + +| Form | Count | Files | Coupled? | +|---|---|---|---| +| Fenced append-only block | 1 | `AGENTS.md` | **No** | +| Whole-file replacement | 1 | `.github/ISSUE_TEMPLATE/config.yml` (upstream's was one line) | **No** | +| Deletion | 2 | the two legacy issue templates | **No** | +| **In-place edit** | **23** | everything else | **Yes, by construction** | + +**So 23 of 27 are exposed to this class, and 4 are not.** That is not a probability estimate — it is a structural statement: an in-place edit *can* be invalidated by upstream restructuring, a fenced block cannot. + +[`AGENTS.md`](https://github.com/launchpad-26/buzz/blob/5d76799d6e44f2f76aa7bd78c5343d339af98f63/AGENTS.md) is the proof of the distinction and it is empirical, not theoretical: it is one of the 8 files both sides touched, and it **auto-merged cleanly in both of the fork's two drops** (#12 and #216), because the fork's change is a delimited append with `` markers. **This is the strongest available argument for [#307](https://github.com/launchpad-26/buzz/issues/307)'s fork-owned-override direction**, and it does not even require a separate file — a fence inside a shared file is enough. + +### Anchor-survival test: nothing else is pending in this drop + +For each in-place edit, checked whether the upstream structure it depends on still exists in `upstream/main`: + +``` + lefthook.yml branch-skew lane anchor PRESENT (1) + lefthook.yml upstream skew script anchor PRESENT (1) + Justfile SIDECARS array anchor PRESENT (3) + Justfile _ensure-sidecar-stubs anchor PRESENT (10) + ci.yml desktop paths-filter group anchor PRESENT (3) + ci.yml file-size step region anchor PRESENT (1) + dev-setup.sh redis guard fn anchor PRESENT (2) + instance-env.sh BUZZ_RELAY_PORT anchor PRESENT (2) + instance-env.sh vite invocation anchor PRESENT (2) + compose.yml BUZZ_IMAGE var anchor PRESENT (1) + Dockerfile OCI source label anchor PRESENT (2) + seed-local-community authority logic anchor PRESENT (3) + buzz-terminal lifecycle io import anchor PRESENT (1) + runtime.rs persona_drift_state anchor PRESENT (1) + restore.rs spawn_agent_child call anchor PRESENT (1) + pack.rs inspect command anchor PRESENT (2) +``` + +All sixteen intact. **No anchor-disappearance break is pending in this drop.** + +**But note carefully what this test does and does not catch.** #360's `relay_url` defect was *not* an anchor disappearance — the anchor (`spawn_agent_child(`) is present, as the table shows. What upstream deleted was a *local binding* used 28 lines away. Anchor survival is necessary and not sufficient, and saying otherwise would be exactly the kind of false assurance this document is about. + +### The two instances on the record + +**Demonstrated (#360): `relay_url`.** Upstream deleted a binding the fork's fix uses; git merged both hunks cleanly; the desktop crate failed to compile. And the resolution that compiles — upstream's own `&key.relay_url` — silently reintroduces the fork's bug, because `ManagedAgentRuntimeKey::new` normalises the URL. **That is the archetype: not merely clean-but-broken, but clean-but-broken with a plausible wrong fix.** + +**Refuted (#360): `bin/.lefthookrc`.** ADR-0022 and #296 both carry this as the worked counter-example. It does not pin lefthook 2.1.3 — it resolves [`bin/lefthook`](https://github.com/launchpad-26/buzz/blob/5d76799d6e44f2f76aa7bd78c5343d339af98f63/bin/lefthook), which the fork points at 2.1.10, and upstream never touched the symlink. Tested in the merged tree: `LEFTHOOK_BIN` resolves and reports `2.1.10`. The mechanism is protective, not hazardous. Only the file's *comment* mentions 2.1.3. + +**So the fork's risk register currently contains one item that is not real and omits the one that is.** + +--- + +## What this means for #273 + +*This section is my recommendation as the author, not a finding. It carries no source reference because no source endorses it: the evidence is above, the judgement is mine.* + +**ADR-0022's declared hole is real, and its size is 23 rows rather than 1 file.** That record says the scope ruling "does not solve that and should not be read as claiming it does" — correct, and now quantified. What changes is the *example*: `.lefthookrc` should be replaced with `relay_url`, which is worse in the way that matters (compiles, plausible wrong fix, no CI lane catches it). + +**Two cheap pre-flights fall out, and neither needs a decision.** Both are mechanical, model-free, and would run in seconds on a drop: + +- **Build the merge before proposing it.** This is the single highest-value change available and it is already how PR #216 was done by hand. The gap is that no automation does it and `merge-tree` is what the PRD's figures rest on. +- **Anchor-survival grep per ledger row.** Requires the ledger to record *what upstream structure each row depends on* — a column [#294](https://github.com/launchpad-26/buzz/issues/294) does not currently have, and the natural companion to the `Upstream-Status` column [#361](https://github.com/launchpad-26/buzz/issues/361) recommends. + +**#296's boundary gets a defensible rule from the research rather than from taste.** Since detection tops out near a third even with dedicated tooling, "the agent may resolve it if a build passes" is not a safe boundary. The `relay_url` case is the proof: a build passes on the wrong resolution. + +**And #307 has its empirical argument.** A fenced append inside a shared file survived two drops untouched while 23 in-place edits remain structurally exposed. That is the cheapest available reduction in this class's size, it costs nothing per drop, and it only governs divergences not yet created — which is why #361 flagged it as getting more expensive to adopt every week. + +--- + +## Revised for the fork's horizon (#357) + +Added after @tucktuck101 decided on 2026-08-22 that the fork ends with the cohort project on +2026-09-17, with no post-cohort maintainer. Two things in this document were sized against a future +that will not arrive. + +**The evidence is unchanged.** The 32% recall figure, the 23-of-27 structural exposure, the +anchor-survival results and the two instances all stand. + +*The rest of this section is my recommendation as the author, not a finding.* + +**"23 of 27 rows are exposed" is still true and much less alarming.** It is a structural statement — +those rows *can* be invalidated by upstream restructuring. With at most a handful of drops left, and +with [#352](https://github.com/launchpad-26/buzz/issues/352) closed on the ground that the 19 +never-conflicted files will very likely never collide, most of that exposure never gets tested. The +number to act on is not 23; it is the 8 files upstream has actually touched, of which 4 conflict. + +**One of my two pre-flights survives and one becomes over-engineering.** *Build the merge before +proposing it* gets stronger: it is cheap, it caught the one real defect, and it is exactly the kind of +demonstrable check a demonstrative target wants. *An anchor-survival grep per ledger row* does not: +it asks #294 to carry a new column describing what upstream structure each row depends on, to guard +rows that will not be touched again. For four live files a person reading the drop covers it. + +**#307's leverage largely collapses, and that is worth saying plainly.** Its argument — express +disagreement as a fenced append rather than an in-place edit — governs only divergences *not yet +created*. With weeks remaining, very few new divergences will be created, so the mechanism has almost +nothing left to govern. The `AGENTS.md` evidence stands as a demonstration that the pattern works; it +no longer supports "adopt this before it gets more expensive", because the compounding it was priced +against has run out. I flagged #307 as high-leverage in +[#361](https://github.com/launchpad-26/buzz/issues/361) on exactly that compounding argument, and I +withdraw that framing. + +**What the class is still worth for.** Not risk management over time, but a worked example: the +`relay_url` defect is a concrete, reproducible instance of a clean merge that compiles and is wrong, +with a plausible wrong fix. That is evidence the cohort understood the failure mode, which is the +deliverable now. + +## Confidence and limits + +**High** on the research findings — figures and quotations are from the paper, and the authors' own conclusion is quoted rather than paraphrased. **High** on the anchor-survival results and the form-based classification: both are reproducible from the commands in this document and in #352. + +**The form-based classification is a structural statement, not a risk estimate.** "23 of 27 are coupled" means 23 *can* be invalidated by upstream restructuring. It does not say how likely that is, and I have no basis for estimating it — nothing in the literature I found gives a per-import rate, and this fork has two drops of history, which is not a sample. + +**Not verified.** The anchor patterns are my own choice of what each edit depends on; a different reading would pick different anchors and could reach a different count. I checked 16 dependencies, not one per in-place edit for all 23 — the remaining ones are the ADR-0005 deployment files and the templates, where the coupling is to values rather than structure. **I did not attempt to detect any third-shape conflict** — an edit that still applies but no longer means what it did — because the literature says the available techniques catch about a third and none of them apply to this codebase without substantial work; that residue remains unmeasured and unmeasurable at this cost. I did not run the fork's test suites at all (disk exhaustion, see #360), so my own second-shape detection was compilation only. I did not read the second and third papers the search surfaced, only the one quoted. I did not investigate structure-aware merge tools or staged-adoption tooling — #366 and #368. + +## Sources + +- [Detecting Semantic Conflicts with Unit Tests](https://arxiv.org/html/2310.02395) — the definition, the SAM tool, recall 0.32 / precision 0.84, and the authors' conclusion +- [Detecting semantic conflicts with unit tests — ScienceDirect](https://www.sciencedirect.com/science/article/pii/S0164121224001158) — journal version of the same work +- [Patches in Electron](https://www.electronjs.org/docs/latest/development/patches) — *"patches can break—sometimes without even a patch conflict or a compilation error"* +- [Using Pre-trained Language Models to Resolve Textual and Semantic Merge Conflicts](https://www.microsoft.com/en-us/research/wp-content/uploads/2022/07/issta22-merge-conflicts-llm.pdf) — the Chromium/Edge downstream-fork framing From b48f919404f0b1d3914e071673b3d3b799599d9e Mon Sep 17 00:00:00 2001 From: tucktuck101 Date: Mon, 24 Aug 2026 15:18:38 +1200 Subject: [PATCH 2/2] docs(research): drop the unsupported precision figure; keep the paper's raw counts (#363) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit arXiv:2310.02395 reports no overall precision for SAM — only per-tool, per-configuration Table 2 cells (0.66-1.0) — and the string 0.84 appears nowhere in it. Deriving one number is also population-dependent (the three false positives are study-wide while the nine detections belong to the best configuration), which is presumably why the authors publish none. The note now cites the verbatim counts only. Co-Authored-By: Claude Fable 5 Signed-off-by: tucktuck101 --- launchpad/Research/363-detecting-clean-but-wrong-merges.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/launchpad/Research/363-detecting-clean-but-wrong-merges.md b/launchpad/Research/363-detecting-clean-but-wrong-merges.md index 327aab9010a..6a5d28781c4 100644 --- a/launchpad/Research/363-detecting-clean-but-wrong-merges.md +++ b/launchpad/Research/363-detecting-clean-but-wrong-merges.md @@ -41,7 +41,9 @@ and the operational version: **The measured effectiveness is the important part.** Evaluating SAM — a tool that generates unit tests with four generators (EvoSuite, Differential EvoSuite, Randoop, Randoop Clean) and runs them against Base, Left, Right and Merge — across 85 change pairs from 51 merge scenarios: - detected **9 of 28 conflicts** — recall **0.32** -- precision **0.84** (3 false positives in 57 non-conflict cases) +- **3 false positives** in 57 non-conflict cases. The paper reports no overall precision for + SAM — only per-tool, per-configuration values in its Table 2 (0.66 to 1.0) — so none is + quoted here. - 19 false negatives, of which manual analysis suggested 13 were theoretically reachable with better test generation The authors' own conclusion: @@ -182,7 +184,7 @@ deliverable now. ## Sources -- [Detecting Semantic Conflicts with Unit Tests](https://arxiv.org/html/2310.02395) — the definition, the SAM tool, recall 0.32 / precision 0.84, and the authors' conclusion +- [Detecting Semantic Conflicts with Unit Tests](https://arxiv.org/html/2310.02395) — the definition, the SAM tool, recall 0.32 (9 of 28 detected), 3 false positives in 57 non-conflict cases, and the authors' conclusion - [Detecting semantic conflicts with unit tests — ScienceDirect](https://www.sciencedirect.com/science/article/pii/S0164121224001158) — journal version of the same work - [Patches in Electron](https://www.electronjs.org/docs/latest/development/patches) — *"patches can break—sometimes without even a patch conflict or a compilation error"* - [Using Pre-trained Language Models to Resolve Textual and Semantic Merge Conflicts](https://www.microsoft.com/en-us/research/wp-content/uploads/2022/07/issta22-merge-conflicts-llm.pdf) — the Chromium/Edge downstream-fork framing