Skip to content

fix: report stale structure baselines, and correct three of them (#227) - #234

Merged
BernardJen merged 1 commit into
mainfrom
fix/structure-check-gradient-and-staleness
Aug 18, 2026
Merged

fix: report stale structure baselines, and correct three of them (#227)#234
BernardJen merged 1 commit into
mainfrom
fix/structure-check-gradient-and-staleness

Conversation

@BernardJen

@BernardJen BernardJen commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Adds the baseline-staleness half of #227. Found three real problems on its first run.

What it found on main

Saver baseline measured ratio
Starfield Warp 0.0011 0.1423 129.4×
Mandelbrot 0.0020 0.0557 27.8×
Voronoi 0.0341 0.1557 4.6×

All three are leftovers from the rewrites in #212, #210 and #211. Each rewrite made its
saver denser; the collapse check only fires on a drop, so nothing complained.

This changes what #227 is about

I named Starfield Warp in that issue, alongside Plasma, as a saver too dim for the check
to protect — its baseline sat under STRUCTURE_MIN_ABS_DROP. That was wrong.

It isn't dim. It measures 0.1423, two orders of magnitude above the margin. Its
baseline was simply stale, left behind by the saver it replaced — thirty white dashes on
navy. Correcting it moves Starfield Warp into full protection and shrinks the known
blind spot from two savers to one
: Plasma, whose 0.0001 is genuine.

A test now asserts Plasma is the only sub-margin saver, so this cannot quietly grow
again.

Reported, not failed

A stale baseline is a defect in the check's configuration, not in the frame under
test. Failing on it would turn every unrelated PR red until somebody fixed a saver they
had never touched. The exit code comes solely from the fail= count — verified by reading
scripts/shader-check.mjs, which matches only SHADERCHECK_DONE ok=(\d+) fail=(-?\d+).

The ratio is calibrated, not picked

4×. I tried 2× first and it flagged two savers whose own measurement is bimodal at
frames 5/12: Raymarch Fractal read 0.0082 in one run and 0.0185 in the next (2.3×),
Reaction Diffusion likewise 2.3×. That is exactly the effect #192 was filed over, and a
threshold has to clear the noisiest saver rather than the median.

4× still catches every real case — Truchet's historical gap was 4.1×, and the three above.

A rejected approach, measured rather than argued

#227 also asks for the flat-gradient case: a smooth ramp passes for a sub-margin saver,
because it has few edges and a wide spread — and so does a healthy Plasma frame.

I tried coarse-stride edge density. A full-frame ramp differs by stride/width between
two samples, so on a 1200-wide frame it scores zero at strides 1, 8 and 32 alike, while
a real image should not. It does separate them — but not by enough to ship:

  • the lowest healthy score is DVD Logo at 0.0026 (stride 8), so any floor that catches
    a ramp sits within 2.6× of a real saver — one saver getting slightly sparser turns CI
    red;
  • it only catches the slowest gradients. A steeper or repeating one scores normally, so it
    would not reliably catch the Physarum-class bug it was aimed at.

Instrumentation removed. That part of #227 stays open, with the negative result recorded
so nobody re-derives it.

Verification

  • Branched from main (7c5aa56).
  • npm run lint clean.
  • npm test — 407 passed (400 + 7 new staleness tests).
  • npm run shadercheck before the baseline fix: 3 stale reported, fail=0.
  • npm run shadercheck after: stale list empty, ok=150 fail=0.
  • All three corrected baselines reproduce on a second run: Starfield 0.1414 vs 0.14,
    Mandelbrot 0.0557 vs 0.055, Voronoi 0.1558 vs 0.15.
  • Baselines re-sorted descending, so chore: re-sort the structure baselines and guard the order with a test #224's ordering test still holds.

Not verified

  • The 4× ratio clears the two noisy savers I know about. If another saver turns out to be
    bimodal by more than 4×, it will produce a spurious warning — visible, and not CI-breaking.
  • The flat-gradient case for Plasma remains open, as above.

The collapse check only ever fires on a DROP, so a baseline sitting far below what
a saver really produces is invisible. Adds a symmetric staleness report and, on its
first run, it found three.

WHAT IT FOUND ON MAIN

  Starfield Warp   baseline 0.0011   measured 0.1423   129.4x
  Mandelbrot       baseline 0.0020   measured 0.0557    27.8x
  Voronoi          baseline 0.0341   measured 0.1557     4.6x

All three are leftovers from the rewrites in #212, #210 and #211. Each rewrite made
its saver denser, the check only fails on drops, so nothing complained.

THIS CHANGES WHAT #227 IS ABOUT

Starfield Warp was named in that issue, alongside Plasma, as a saver too dim to
protect -- its baseline sat under STRUCTURE_MIN_ABS_DROP. That was wrong. It is not
dim: it measures 0.1423, two orders of magnitude above the margin. Its baseline was
simply stale, left behind by the saver it replaced (thirty white dashes on navy).

Correcting it moves Starfield Warp into full protection and shrinks the known blind
spot from two savers to one -- Plasma, whose 0.0001 is genuine. A test now asserts
that Plasma is the only sub-margin saver, so this cannot quietly grow again.

REPORTED, NOT FAILED

A stale baseline is a defect in the check's configuration, not in the frame under
test. Failing on it would turn every unrelated PR red until somebody fixed a saver
they had not touched. The exit code comes solely from the fail= count, which this
does not touch -- verified against scripts/shader-check.mjs.

THE RATIO IS CALIBRATED, NOT PICKED

4x. 2x was tried first and flagged two savers whose own measurement is bimodal at
frames 5/12: Raymarch Fractal read 0.0082 in one run and 0.0185 in the next, and
Reaction Diffusion likewise 2.3x. That is the effect #192 was filed over, and the
threshold has to clear the noisiest saver rather than the median. 4x still catches
every real case -- Truchet's historical gap was 4.1x, and the three above.

A REJECTED APPROACH, MEASURED

#227 also asks for the flat-gradient case: a smooth ramp passes for a sub-margin
saver, because it has few edges and a wide spread, and so does a healthy Plasma
frame. Coarse-stride edge density was tried -- a full-frame ramp differs by
stride/width between samples, so it scores zero at strides 1, 8 and 32 alike, while
a real image should not. It does separate them, but not by enough to ship: the
lowest healthy score is DVD Logo at 0.0026 (stride 8), so any floor that catches a
ramp sits within 2.6x of a real saver, and it only catches the slowest gradients --
a steeper one scores normally. Instrumentation removed. That part of #227 stays
open.
@BernardJen
BernardJen merged commit 76b8259 into main Aug 18, 2026
1 check passed
@BernardJen
BernardJen deleted the fix/structure-check-gradient-and-staleness branch August 18, 2026 08:41
BernardJen added a commit that referenced this pull request Aug 18, 2026
…heck (#235) (#251)

A flat gradient passed for Plasma. Edge density could not see it -- Plasma's
baseline is 0.0001, so a drop to zero is smaller than the absolute margin -- and
#229's spread check catches a UNIFORM frame, not a smooth ramp. To both of those
statistics a gradient and a healthy Plasma frame are the same picture: near-zero
edges, wide luminance spread.

WHAT SEPARATES THEM

A gradient is MONOTONIC. Its signed slope along a row never reverses; a real image,
however smooth, reverses constantly. Measured across all 30 savers:

  healthy Plasma        0.5948
  radial gradient       0.0586   (one reversal per row)
  linear ramp           0.0000   (none, by construction)

Threshold 0.12: a 5.0x margin under the healthy value and 2x above the radial case.

TWO STRIDES, AND THE MAX

Falling Sand reads 0.5833 at stride 4 and 0.0000 at stride 16 -- its grains are
finer than the coarse step, so one stride alone would call a sparse saver monotonic.
Taking the max of strides 4 and 16 fixes that.

GATED TO SUB-MARGIN SAVERS, WHICH IS PLASMA ALONE

Not applied set-wide, and that is a deliberate limit rather than laziness. The
lowest undulation among savers with a non-zero baseline is Julia Set at 0.1544,
which clears 0.12 by only 1.3x. A set-wide floor would therefore sit within a third
of a healthy saver -- the same thin margin that made the coarse-stride approach
unshippable in #234, and no more acceptable here.

VERIFIED BY FAULT INJECTION, NOT BY ELIMINATION

Each case names the rule that fired:

  linear ramp      fail=5  "frame is monotonic: undulation 0.0000 is below 0.12"
  radial gradient  fail=5  "frame is monotonic: undulation 0.0586 is below 0.12"
  uniform grey     fail=5  caught by the #229 spread rule, as intended
  clean            ok=150 fail=0, Plasma undulation 0.5948

A SELF-INFLICTED ERROR WORTH RECORDING

The first measurement read 0.0000 for all thirty savers, which looked like a
finding and was a bug: the block sat AFTER lums.sort(), and a sorted array is
monotonic by definition. The file already warned about this for edge density. The
computation is now before the sort, with a comment saying why it must be.

AND A HARNESS BUG FOUND WHILE VERIFYING

The failure reason never reached CI output. scripts/shader-check.mjs extracts
console lines with a per-line regex against Chrome's INFO:CONSOLE(n] "..." format,
so a MULTI-LINE message is dropped entirely -- its first line has no closing quote
and the rest have no prefix. The FAIL log was multi-line, so every failed run has
been printing "FAILING SAVERS: <name>" with no reason attached, and the reason could
only be recovered by reading the page. Now emitted on one line.

That bug is why this commit can say which rule caught which fault instead of
inferring it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant