Skip to content

✨ Detect and correct global 2π branch errors in MCPC-3D-S - #32

Merged
vanandrew merged 13 commits into
mainfrom
fix/branch-selection-residual
Jul 28, 2026
Merged

✨ Detect and correct global 2π branch errors in MCPC-3D-S#32
vanandrew merged 13 commits into
mainfrom
fix/branch-selection-residual

Conversation

@vanandrew

@vanandrew vanandrew commented Jul 28, 2026

Copy link
Copy Markdown
Owner

Replaces a cascade of five hardcoded field-magnitude thresholds with a two-stage rule, and fixes a real 40 Hz field-map artifact found on OpenNeuro data.

The bug

Spatial unwrapping recovers phase differences between voxels, never the absolute turn count, so the dual-echo field is determined only modulo 1/dTE. ROMEO's correct_global resolves that by preferring the smallest field, which is equivalent to requiring |f| < 1/(2·dTE).

It re-derives that choice on every frame, from an unweighted median of rounded wrap counts over a dilated mask. When a subject's field approaches the half-wrap the ballot sits on a knife edge, and ordinary frame-to-frame variation tips it. Because the turn count is an integer, a tipped frame is displaced by a full wrap rather than slightly.

Two subjects in ds006131 (TEs 14.2/38.93/63.66 ms, wrap 40.44 Hz, half-wrap 20.22 Hz) do this:

                     correct_global alone              with the selector
sub-24630   -24.69..+17.26 Hz, 41.44 Hz max step   +14.13..+19.17 Hz, 5.04 Hz max step
            158 of 243 frames displaced             0 frames displaced
sub-20828   -23.74..+17.78 Hz, 40.98 Hz max step   +15.24..+18.63 Hz, 1.54 Hz max step
             19 of 243 frames displaced             0 frames displaced

The damage is temporal. A uniformly wrong turn count would be a constant offset in a quantity already referenced to an arbitrary demodulation frequency, and undetectable. A count that varies between frames of one acquisition corrupts the temporal structure the acquisition exists to measure.

The rule

  1. Consistency. A turn-count error of M shifts the phase offset by wrap(2π·k·M), k = TE0/dTE, landing as the same additive constant on every echo. Fitting a line through the echoes and reading its intercept measures it: a self-consistent reconstruction extrapolates through the origin. The rejection scale is analytic, |wrap(2π·k)|, so nothing is tuned to data. Fitting and non-fitting candidates separate by 6–9 orders of magnitude.
  2. Prior, only on a tie. Several candidates can be exactly through-origin; that alias is real and no phase statistic resolves it. Among survivors, take the smallest weighted-median field.

This is detectable at all only because the offset and the field are computed in separate stages. The offset is formed first; the field comes from a second unwrap that applies its own global correction with no knowledge of which count the offset assumed. On a tipped frame the two disagree, and that mismatch is what the intercept measures.

Stage 2 applies the same prior as correct_global, on a better-conditioned statistic: the field itself, weighted, over an eroded mask, rather than rounded counts over a dilated one. On the failing frames both candidates score ~1e-7, so stage 1 detects the flip but cannot rank them; the prior settles it.

The weighting

The prior weights by mag1², the second echo. The field is read from phase1 − phase0, whose noise is dominated by the weaker echo; weighting on mag0 gives full weight to fast-T2* voxels (healthy echo 0, collapsed echo 1), which are exactly the air–tissue interfaces the prior should discount.

This does not move the decision boundary, which is fixed at 1/(2·dTE), and barely moves the margin. What it buys is stability, and that is what predicts a flip:

total errors worst-case margin between-subject spread
mag0² 0 0.884 Hz 2.044 Hz
mag1² 0 1.052 Hz 0.667 Hz

Also documented in the docstrings: the estimator is a median rather than a mean or M-estimator because the field distribution is broad and right-skewed with the half-wrap boundary inside its body, so anything pulled toward the mean reads closer to the boundary without being more accurate. A Huber estimator cut the worst-case margin to 0.045 Hz; a half-sample mode was tried and reverted for discontinuity.

Changes

  • Add _evaluate_branch, _branch_intercept_step, _weighted_median, _select_branch; remove _select_branch_cascade and all four heuristic constants.
  • Return 0 when TE0/dTE is an integer, where the count cannot move the offset.
  • Correct two annotations: echo times are numpy scalars at every call site, not arrays; rescale_phase's min/max are floats.
  • 16 tests, including a regression fixture built from the real failing frames (tests/data/branch_flip/, 9.7 MB, excluded from the sdist). Cropped to the brain bbox; deliberately not decimated, since resampling tips the knife-edge ballot and would invert what the test checks.

Breaking

--wrap-limit is removed from wk-medic and wk-unwrap-phase, and wrap_limit= from medic() and unwrap_and_compute_field_maps(). It existed to disable the heuristics this PR deletes.

Validation

100 runs, 24,105 frames, 10 OpenNeuro ME-EPI datasets, six TE0/dTE ratios (0.51–0.78) and echo spacings 16.2–27.5 ms. Zero errors under either weighting.

71 of those runs were processed after the weighting was chosen and held out from the comparison. mag1²'s spread advantage was 2.3× on the data that selected it and 2.7× on the held-out set, so it replicated rather than regressing.

The prior changes an answer on 2 of 77 prior-live runs, both above |f|/half-wrap = 0.81; below 0.59 it was never load-bearing. The worst frame anywhere reached 0.948, leaving a 2.10 Hz decision margin on a 40.44 Hz wrap.

CI

Pure Python — no pybind11, ITK, or wheel-matrix impact. Cost is ~+14% per frame for three candidate reconstructions.

The global 2*pi branch for the MCPC-3D-S phase offset was picked by a
cascade of five hardcoded field-magnitude thresholds, with a --wrap-limit
escape hatch for when the cascade made things worse. Replace it with a
criterion that has no tuned constants.

A branch error of M wraps shifts the phase offset by wrap(2*pi*k*M) with
k = TE0/dTE, and that lands as the same additive constant on every echo.
Fitting a line through the two echoes and reading its intercept measures
that constant directly: the correct branch extrapolates through the
origin, a wrong one does not. The rejection scale is analytic --
|wrap(2*pi*k)| -- so nothing is fit to data.

- Add _evaluate_branch / _branch_intercept_step / _select_branch; drop
  _select_branch_cascade, FMAP_PROPORTION_HEURISTIC,
  FMAP_AMBIGUIOUS_HEURISTIC, BRANCH_SELECTION_MODES and
  BRANCH_CONSISTENCY_RATIO.
- Act only when exactly one candidate is consistent. On a tie the
  branches are a genuine alias that no phase statistic can resolve, so
  defer to ROMEO's correct_global rather than guess: a measured
  centre-frequency sweep put a smallest-|field| tiebreaker at 12% better
  and 12% worse, including cases where correct_global had been right.
- Return 0 when TE0/dTE is an integer, where the branch cannot move the
  offset at all.
- Drop wrap_limit from medic(), unwrap_and_compute_field_maps() and the
  wk-medic / wk-unwrap-phase CLIs. BREAKING: --wrap-limit is gone; it
  existed to disable the heuristics this commit removes.
- Correct two annotations the new tests exposed: echo times are numpy
  scalars at every call site, not arrays, and rescale_phase's min/max
  are floats, not ints.
- Add 13 tests covering the analytic step, the decision rule, and
  end-to-end recovery of an injected wrap on the bundled data.

Validated on 14090 frames across 58 runs from eight OpenNeuro datasets
spanning six TE0/dTE ratios (0.51-0.78). No frame selected a non-zero
branch and the largest branch-0 intercept was 1.9e-07 of the cutoff, so
the output is unchanged on all data tested. Fitting and non-fitting
branches separate by six to nine orders of magnitude when the rule does
fire, verified by injecting known wraps.
Copilot AI review requested due to automatic review settings July 28, 2026 03:12
@codecov

codecov Bot commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.65217% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.25%. Comparing base (764aa57) to head (d066cf7).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
warpkit/unwrap.py 95.65% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #32      +/-   ##
==========================================
+ Coverage   95.01%   96.25%   +1.23%     
==========================================
  Files          18       18              
  Lines        1265     1280      +15     
==========================================
+ Hits         1202     1232      +30     
+ Misses         63       48      -15     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates MEDIC’s MCPC-3D-S global (2\pi) branch selection by replacing the prior field-magnitude heuristic cascade with an intercept-based consistency test (no tuned constants), and removes the now-obsolete wrap_limit API/CLI surface. It also adjusts a couple of type annotations and adds targeted tests to validate the new decision rule and an end-to-end injected-wrap recovery case.

Changes:

  • Implement intercept-based branch evaluation/selection (_evaluate_branch, _branch_intercept_step, _select_branch) and remove the prior heuristic cascade and its tuning constants.
  • Remove --wrap-limit / wrap_limit= from CLIs and library entry points that previously exposed heuristic disabling.
  • Add tests for the analytic intercept step, decision-rule branches, and end-to-end recovery on bundled data.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
warpkit/utilities.py Adjusts rescale_phase parameter annotations (int → float).
warpkit/unwrap.py Adds intercept-based branch selection and removes heuristic-based wrap limiting.
warpkit/scripts/unwrap_phase.py Removes --wrap-limit plumbing from wk-unwrap-phase.
warpkit/scripts/medic.py Removes --wrap-limit plumbing from wk-medic.
warpkit/distortion.py Removes wrap_limit from the public medic() wrapper call into unwrapping.
tests/test_unwrap.py Adds unit tests for the new branch selector and an end-to-end injected-wrap recovery test.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread warpkit/utilities.py
Comment thread warpkit/unwrap.py
Comment thread tests/test_unwrap.py Outdated
Addresses review feedback on #32 and unblocks the lint job.

- `mcpc_3d_s` documents `te0`/`te1` as `np.float32 | float` to match the
  scalar signature; `rescale_phase` documents `min`/`max` as floats.
- Split `test_branch_selector_noop_when_nothing_fits` in two. The all-zero
  intercepts case is an all-way perfect fit that collapses the observed scale,
  not a failed fit, so it gets its own test and an accurate docstring.
- Reformat the README's Python block: ruff 0.16 formats code in markdown, which
  fails `ruff format --check` on main as well as here.
Restores the smallest-|field| tiebreaker removed earlier in this branch. The
evidence for removing it came from an induced centre-frequency sweep, which
only exercises the regime where the prior is meant to lose -- a genuine global
field offset. It never covered the opposite regime, an artifactual one-wrap
flip, and that is the one that actually occurs.

Found on ds006131 sub-20828 (TEs 14.2/38.93/63.66 ms, k=0.5742, wrap 40.44 Hz).
That subject's global field sits at 17.7 Hz against a 20.22 Hz half-wrap
boundary, so correct_global's median of rounded wrap counts is on a knife edge
and tips over on individual frames: its ballot reads 0, +1, 0, -1 frame to
frame, and each tip flips the dual-echo field a full wrap for one frame.

The intercept test detects the flip -- the fitting set moves from {-1, 0} to
{0, +1} -- but both survivors score ~1e-7, so it cannot rank them. The prior
can, and correctly: branch +1 restores +17.7 Hz against branch 0's -22.8 Hz.
It is the same rule correct_global applies, on a better conditioned statistic
(magnitude-weighted, eroded mask, on the field itself rather than an unweighted
median of rounded counts over a dilated mask), which is what makes it stable
where correct_global is not.

Full pipeline on that run, before and after:

    before   field +16.46..+53.36 Hz   max step 35.939 Hz   34 wrap-sized steps
    after    field +16.38..+19.62 Hz   max step  1.462 Hz    0 wrap-sized steps

19 of 243 frames were single-frame ~40 Hz excursions; all are gone, and the
224 healthy frames are untouched.

- Restore _weighted_median and the tie branch of _select_branch.
- Rewrite the docstring around this measurement instead of the sweep.
- Replace the two deferral tests with ones built from the real frame-44
  intercepts and fields.
Pins the failure that motivated keeping the field prior, using the real data
rather than hand-written numbers.

Two frames from ds006131 sub-20828 run-01 (TEs 14.2/38.93/63.66 ms, k=0.5742,
wrap 40.44 Hz): the original frame 43, which is healthy, and frame 44, one of
19 frames out of 243 where correct_global's ballot tips and the dual-echo field
jumps a full wrap. The selector must leave the first alone and return +1 on the
second.

- tests/data/branch_flip/ (9.7 MB, excluded from the sdist by the existing
  tool.scikit-build exclude of tests/data).
- Cropped to the brain bounding box, which reproduces the behaviour exactly.
  Deliberately NOT decimated: correct_global's median of rounded wrap counts is
  on a knife edge here, so resampling tips it the other way -- 2x decimation
  makes the flip vanish on frame 44 and appear on the healthy frame 43. That
  would silently invert what the test checks.
- Also pins that the broken frame is a genuine {0, +1} tie on the intercept, so
  a classify-only rule provably cannot fix it and the prior is load-bearing.
@vanandrew vanandrew changed the title ♻️ Replace phase-branch heuristics with an intercept test ✨ Detect and correct global 2π branch errors in MCPC-3D-S Jul 28, 2026
The tie-breaking prior asks which candidate branch puts the global field
closest to zero. It was estimating that level with a magnitude-weighted median
over an eroded brain mask. A median is the wrong statistic for this quantity.

The field distribution over a brain is a sharp peak at bulk-tissue value with
long tails from sinuses, dropout and edges. The scanner's frequency adjustment
centres that peak near zero -- not the median-including-tails -- so a median
estimates something the shim never set, and is pulled toward whichever tail is
heavier. That is the same defect that makes correct_global's own estimate tip
over: its dilated mask simply has more tail.

Margins to the +/-1/(2*dTE) flip boundary, ds006131 sub-20828, worst of the
healthy/broken frame pair:

    correct_global (median, dilated)      -0.26 Hz   flips
    median, eroded                         1.25 Hz
    magnitude-weighted median, eroded      2.57 Hz   (previous)
    half-sample mode, eroded              10.27 Hz   (this commit)

Across all 243 frames of that run the two estimators pick identically -- same
18 frames corrected, zero disagreements -- and the full pipeline output is
unchanged, but the worst-case margin goes from 1.51 Hz to 9.34 Hz.

The half-sample mode takes the shortest interval containing half the remaining
points, recursively. It has no bin width and no bandwidth, so it introduces no
tuned constant; a binned mode agrees with it to within 9.8-11.2 Hz of margin
across a 64x range of bin counts, confirming the choice is not load-bearing.
It is also cheaper than the weighted median it replaces (2.8 ms vs 6.9 ms at
120k voxels) because it never sorts a weight array.
Derivation only -- no dataset names, measurements or validation numbers. Every
quantity is a function of the echo times or the signal model, so it stays valid
independent of what we happen to have tested against.

Covers:

- The signal model and the two derived scales, wrap spacing 1/dTE and the
  half-wrap 1/(2*dTE) that any smallest-|f| prior decides against.
- The two ambiguities, each with a proof. The exact alias
  (theta - W(2*pi*k*N), f + N/dTE) reproduces the wrapped data identically and
  is therefore undetectable; an offset-branch mismatch instead lands a constant
  W(2*pi*k*M) on every echo and is.
- Why extra echoes buy nothing: with even spacing every pair aliases at the same
  period, so five echoes carry as much branch information as two.
- The intercept criterion, and why its rejection scale |W(2*pi*k)| is analytic
  rather than tuned.
- Why the intercept and the field prior are both required -- the intercept is
  blind to a sign-preserving shift of the field, the prior to a sign-reversing
  one, so neither is sufficient alone.
- Why a median is the wrong estimator for the prior. Writing the field
  distribution as (1-eps)*p_bulk + eps*p_tail, the mode is consistent for the
  bulk centre while the median is not, and mask dilation raises eps. The
  half-sample mode estimates the mode without a bin width or bandwidth.
- Structural limits: ambiguity A is unaddressable by construction, dTE is
  bounded below by the EPI readout and above by T2*, the two ultra-high-field
  effects act in opposite directions, and a whole-turn shift of a multi-echo
  unwrapper's template echo is a pure slope change that no intercept test can
  see.

Source is committed alongside the PDF so it stays regenerable with
`typst compile notes/branch-selection-theory.typ`.
The first draft was written from the inside out: it opened on "branch
selection", which presumes the reader already knows what a branch is and why
one would be selecting among them. Rewritten for someone meeting the problem
for the first time, and organised around the ambiguities rather than around the
procedure that resolves them.

The spine is now a taxonomy. Removing an estimated offset leaves a residual
psi_e that should be a straight line through the origin; fitting it gives an
intercept c and a slope s, and every possible error is a displacement in that
(c, s) plane. Exactly three cases matter:

  A  offset and field both wrong, in a matched way   -> data unchanged, invisible
  B  offset wrong alone                              -> constant c, detectable
  C  field wrong alone                               -> pure slope, c stays zero

Branch selection then falls out as the answer to B rather than being the
premise, and the limits of the approach are visible as the cases the taxonomy
already accounts for. Ambiguity C -- a whole-turn error in a multi-echo
unwrapper's template echo, which propagates proportionally to echo time -- is
promoted from a footnote to a peer of the other two, since it is exactly the
failure an intercept test cannot see.

Also opens with what phase is and why it is ambiguous at all, so no pipeline
knowledge is assumed, and renames the file to match the new framing.
- Cut the gradient-echo primer. The reader is assumed to know what MR phase is;
  what needs motivating is why the coil offset has to be recovered at all. The
  opening now makes that case directly: field mapping wants the rate of phase
  accumulation, theta is the constant it accumulates from, and theta cannot be
  ignored because it varies rapidly in space and so breaks the smoothness that
  spatial unwrapping depends on.
- Drop the now-redundant restatement of theta in the signal model.
- Byline and date; author metadata instead of the project name.
Reverts the field-prior estimator to a magnitude-weighted median. The mode
regressed a run that the median handled correctly, caught by the corpus scan
before this left the branch.

On ds005354 sub-02 run-00 (k=0.5089, wrap 36.35 Hz) the mode made 108 of 223
frames select a non-zero branch, producing a 32.7 Hz step in the field time
series. The median selects branch 0 on all 223 and the run is flat. That is the
only run in the corpus scan with a wrap-sized step under either estimator, and
the median baseline has none.

The cause is the assumption the mode carries. That subject's field distribution
is bimodal -- peaks 21 Hz apart at 95% and 100% of maximum -- so the argmax
flips on noise between adjacent frames. Magnitude weighting does not separate
peaks that close (they stay within 5%), and restricting to high-SNR voxels made
it worse, swinging between the two peaks on consecutive frames.

The general lesson, now recorded on the function: continuity matters more here
than freedom from bias. The estimate feeds a threshold comparison against
1/(2*dTE) that must be stable frame to frame, and a median is a continuous
functional of the data where any argmax is not. The earlier claim of a 4x
margin improvement came from a single subject and did not generalise -- the
mode-minus-median offset is subject-specific in sign as well as magnitude.

A continuous alternative that avoids both defects, scoring candidates by field
mass at zero, was also tested. It reproduces the median's decisions on both
stress cases but does not improve on them, so it was not adopted either.

Also drops the mode material from notes/phase-offset-ambiguities.typ and
replaces it with the estimator actually used, plus the continuity requirement
that rules the alternatives out.
It was never observed, and the derivation behind it was wrong.

The claim was that a whole-turn error in a multi-echo unwrapper's template echo
propagates to echo e scaled by t_e/t_0, i.e. proportionally to echo time, and
that a proportional displacement is a pure slope change with exactly zero
intercept -- so no intercept test could ever see it.

The proportional step does not survive the implementation. Non-template echoes
are unwrapped by unwrap_voxel (include/romeo/algorithm.h), which returns
new - 2*pi*round((new - old)/2*pi): the correction is a whole number of turns.
Shifting the reference therefore shifts echo e by 2*pi*round(t_e/t_0) turns, not
2*pi*t_e/t_0. For TEs 14.2/38.93/63.66 those ratios are 2.742 and 4.483, which
round to 3 and 4 -- not proportional to echo time, so the intercept is not zero
and an intercept test would in fact partially see it.

Nor was it ever seen in data. Whole-run scans counted frame-to-frame steps at
1000/TE0 Hz, its predicted signature, and found none on any run of any protocol.
The one supporting measurement I had -- the template echo's pre-correction wrap
count going non-zero on some frames -- shows correct_global doing its job rather
than failing at it, and is not evidence of the failure at all.

The taxonomy is now the two cases the propositions actually establish: offset
and field wrong together (invisible), and offset wrong alone (a constant on
every echo). Both are proved, and B is the one the selector acts on.
The dual-echo field is read from phase1 - phase0, whose noise is dominated
by the weaker echo. Weighting on mag0 gave full weight to fast-T2* voxels,
a healthy echo 0 alongside a collapsed echo 1, which are exactly the
air-tissue interfaces the prior should discount.

Measured over 100 runs and 24k frames from ten OpenNeuro ME-EPI datasets,
71 of them held out from the comparison that chose the weighting. The
decision boundary is fixed at 1/(2*dTE) and does not move, and the
worst-case margin barely changes (0.884 -> 1.052 Hz on the tightest
subject). What improves is stability: the estimator's offset varies by
0.67 Hz across subjects under mag1**2 against 2.04 Hz under mag0**2, and
that between-subject spread is what predicts a wrap flip. Zero errors
under either weighting.

Also record in the docstrings why the estimator is a median rather than a
mean or M-estimator: the field distribution is broad and right-skewed and
the half-wrap boundary falls inside its body, so anything pulled toward
the mean reads closer to the boundary without being more accurate.
The note is independent of the selector change and is going up separately
on docs/phase-offset-note. Keeps this PR to the code change.
@vanandrew
vanandrew merged commit f462497 into main Jul 28, 2026
22 checks passed
@vanandrew
vanandrew deleted the fix/branch-selection-residual branch July 28, 2026 16:22
vanandrew added a commit to vanandrew/sdcflows that referenced this pull request Jul 29, 2026
warpkit 1.5.0 removes the ``wrap_limit`` argument from
``warpkit.api.unwrap_phase``, so forwarding it raised

    TypeError: unwrap_phase() got an unexpected keyword argument 'wrap_limit'

inside the ``unwrap`` node, failing test_dynamic_unwarp_run on both
fixtures.

The flag only existed to disable a cascade of hardcoded field-magnitude
thresholds, which warpkit replaced with an automatic 2-pi branch
selector (vanandrew/warpkit#32). There is no successor input to thread
through, so the trait is dropped rather than renamed. Nothing in
SDCFlows ever set it.
vanandrew added a commit to vanandrew/sdcflows that referenced this pull request Aug 9, 2026
warpkit 1.5.0 removes the ``wrap_limit`` argument from
``warpkit.api.unwrap_phase``, so forwarding it raised

    TypeError: unwrap_phase() got an unexpected keyword argument 'wrap_limit'

inside the ``unwrap`` node, failing test_dynamic_unwarp_run on both
fixtures.

The flag only existed to disable a cascade of hardcoded field-magnitude
thresholds, which warpkit replaced with an automatic 2-pi branch
selector (vanandrew/warpkit#32). There is no successor input to thread
through, so the trait is dropped rather than renamed. Nothing in
SDCFlows ever set it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants