Graded 30-case golden benchmark (easy/medium/hard, multi-channel hard)

.claude/skills/codameter-advisor/references/golden_datasets.md

@@ -1,35 +1,40 @@
 # Golden datasets
 
-Seeded synthetic CCF suites with known ground-truth dv/v(t), covering the
-mainstream per-application cases and four edge regimes. Two consumers: the pytest
-regression oracle (`tests/test_golden.py`) and this advisor's live validation.
+Seeded synthetic CCF suites with known ground-truth dv/v(t), organised as a
+**graded benchmark**: 30 cases, 10 per difficulty grade, spanning the monitoring
+applications. Two consumers: the pytest regression oracle
+(`tests/test_golden.py`) and this advisor's live validation.
 
 ## Layout
 
 - `tests/data/golden/manifest.json`: the committed oracle: one entry per case
-  with its recipe (use case, years, snr, seed, cadence, decorr) and the frozen
-  expected metrics (baseline-aligned RMS, plus any probes). Version-controlled.
+  with its recipe (grade, use case, motif, snr, seed, channels, decorr) and the
+  frozen expected metrics (baseline-aligned RMS). Version-controlled.
 - `tests/data/golden/cache/*.npz`: the regenerated arrays. Deterministic from
   the seed, so they are gitignored, not committed.
 - `codameter.golden`: the generator. `CASES` is the recipe list; `generate(id)`
-  rebuilds arrays; `regenerate_manifest()` recomputes the expected metrics.
+  rebuilds arrays; `recover(d, cfg, eps)` runs the pipeline (aggregating channels
+  for multi-channel cases); `regenerate_manifest()` recomputes expected metrics.
 
-## The cases
+## The grades
 
-Mainstream (one per application): `volcano_mainstream`, `earthquake_mainstream`,
-`landslide_mainstream`, `groundwater_mainstream`, `cryosphere_mainstream`,
-`geothermal_mainstream`. Each should recover its truth to well under 0.2 % RMS.
+Case ids are `{grade}-{application}-{nn}`, e.g. `easy-volcano-01`,
+`hard-groundwater-08`. Each grade cycles through the applications (volcano,
+earthquake/fault, landslide, groundwater, cryosphere, geothermal).
 
-Edge:
-- `low_snr_large_dvv`: SNR ~2 with a several-percent pre-failure drop; the
-  cycle-skipping regime that splits stretching from cross-spectral methods.
-- `clock_drift_seasonal`: a growing clock error plus a seasonal late-coda warp;
-  injects a spurious dv/v (Zhan 2013 / Daskalakis 2016).
-- `freqdep_shallow_deep`: shallow (high-freq) and deep (low-freq) layers carry
-  different dv/v; the band selects which one you recover. Has a probe proving the
-  deep band recovers the deep layer.
-- `sparse_decorr`: every-third-day sampling with 30 % waveform decorrelation;
-  stresses the reference and stacking warm-up.
+- **easy** (split `validation`): a pure seasonal signal at high SNR (8-12),
+  single channel. Best-practice recovery should be well under 0.2 % RMS.
+- **medium** (split `validation`): a transient coseismic-style drop with
+  logarithmic partial healing, plus more measurement noise (SNR 3-5).
+- **hard** (split `test`): a **multi-channel** (4-channel) problem whose truth
+  combines a transient drop-and-heal, a full hydrological seasonal cycle, and a
+  long-term trend, at low SNR (2-4) with waveform decorrelation. Channels are
+  measured independently and aggregated (`golden.recover`).
+
+Note: these synthetics impose a spatially homogeneous dv/v, so recovery is
+insensitive to the band/window as long as the band overlaps the coda's content.
+The benchmark grades estimator, reference, stacking, and aggregation robustness
+under noise and complexity, not depth-band selection.
 
 ## Inspect
 

.claude/skills/codameter-advisor/references/validation_loop.md

@@ -8,8 +8,8 @@ every difference is an artifact of the processing choice, not of nature.
 ## The primitives
 
 - `codameter.golden.generate(case_id)` returns `{ccfs, t, days, truth, fs, ...}`
-  for a seeded case. `codameter.golden.MAINSTREAM_BY_USE_CASE[use_case]` gives the
-  matched mainstream case id.
+  for a seeded case. `codameter.golden.MAINSTREAM_BY_USE_CASE[use_case]` gives a matched
+  (easy-grade) case id for an application.
 - `codameter.deviations.run_pipeline(ccfs, t, fs, cfg, eps_max=...)` returns
   `(dvv, valid)` for one config.
 - `codameter.golden._rms(dvv, truth, days, valid)` is the baseline-aligned RMS
@@ -79,7 +79,7 @@ bar comes from marginalising the processing choice:
 pixi run python - <<'PY'
 from codameter import golden
 from codameter.uq_bayes import bayes_dvv_from_ccfs
-d = golden.generate("volcano_mainstream")
+d = golden.generate("easy-volcano-01")
 res, ens = bayes_dvv_from_ccfs(d["ccfs"], d["t"], d["fs"],
                                truth=d["truth"], days=d["days"], cadence=4)
 import numpy as np
@@ -95,7 +95,7 @@ it when the user cares about the propagated uncertainty, and say it is running.
 - A lower RMS is better recovery; state the percentage, not an adjective.
 - A `moving` reference erases the slow trend, so on a trend case it shows a large
   RMS by design. That is the point, not a bug.
-- Edge cases (low SNR, clock drift, freqdep, sparse) are in the golden manifest.
-  Pull the matching one with `golden.generate(<id>)` if the user's situation is
-  an edge case rather than a mainstream one; ids are in
-  `golden.CASES_BY_ID`.
+- Harder cases (medium = transient + noise; hard = multi-channel composite)
+  are in the golden manifest; pull one with `golden.generate("<grade>-<app>-<nn>")`
+  when the user's situation is noisier or more complex than a clean seasonal one.
+  Ids are in `golden.CASES_BY_ID`.

scripts/plot_golden.py

@@ -0,0 +1,126 @@
+#!/usr/bin/env python3
+"""Materialize and visualize the graded golden synthetic dv/v datasets.
+
+For every case in :mod:`codameter.golden` this generates the arrays (writing the
+seeded ``.npz`` to the golden cache) and saves a verification figure with three
+panels:
+
+  (a) the reference coda CCF trace, to judge waveform realism (band-limited,
+      decaying coda; measurement window shaded);
+  (b) the daily CCF gather (day vs lapse time; the channel-mean for multi-channel
+      hard cases), to see coherence, noise level, and signal;
+  (c) the ground-truth dv/v(t) with the series recovered by the recommended
+      pipeline (via golden.recover, so multi-channel cases are aggregated),
+      baseline-aligned, RMS annotated.
+
+Run:  pixi run python scripts/plot_golden.py [--outdir DIR]
+"""
+from __future__ import annotations
+
+from pathlib import Path
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+
+from codameter import golden
+from codameter import use_cases as uc
+
+PCT = 100.0
+OUTDIR = Path(__file__).resolve().parents[1] / "tests" / "data" / "golden" / "figs"
+GRADE_C = {"easy": "#2e7d32", "medium": "#e08a00", "hard": "#c62828"}
+C = {"truth": "#20222b", "rec": "#c62828", "win": "#f2c14e"}
+
+
+def _align(rec, truth, days, valid, frac=0.2):
+    v = valid & np.isfinite(rec)
+    out = np.full_like(rec, np.nan)
+    out[v] = rec[v]
+    if v.sum() < 5:
+        return out
+    cut = np.quantile(days[v], frac)
+    base = v & (days <= cut)
+    if base.sum() >= 2:
+        out[v] = out[v] - np.mean(out[base]) + np.mean(truth[base])
+    return out
+
+
+def plot_case(case_id: str, outdir: Path = OUTDIR) -> Path:
+    recipe = golden.CASES_BY_ID[case_id]
+    app = recipe["use_case"]
+    d = golden.generate(case_id)                      # materialize arrays
+    t, days, ccfs, fs = d["t"], d["days"], d["ccfs"], d["fs"]
+    cfg = uc.recommend(app, **recipe.get("config", {}))   # target band for depth cases
+    band, window = cfg["band"], cfg["window"]
+
+    fig = plt.figure(figsize=(11, 7.2))
+    gs = fig.add_gridspec(2, 2, height_ratios=[1.0, 1.05], hspace=0.42, wspace=0.26)
+    axA, axB = fig.add_subplot(gs[0, 0]), fig.add_subplot(gs[0, 1])
+    axC = fig.add_subplot(gs[1, :])
+
+    ref = ccfs[: int(0.6 * len(ccfs))].mean(axis=0)
+    axA.plot(t, ref, lw=0.6, color=C["truth"])
+    for s in (+1, -1):
+        axA.axvspan(s * window[0], s * window[1], color=C["win"], alpha=0.25, lw=0)
+    axA.set(xlabel="lapse time (s)", ylabel="amplitude", title="(a) reference coda CCF")
+    axA.margins(x=0)
+
+    vmax = np.percentile(np.abs(ccfs), 99)
+    axB.imshow(ccfs, aspect="auto", cmap="seismic", vmin=-vmax, vmax=vmax,
+               extent=[t[0], t[-1], days[-1], days[0]], interpolation="nearest")
+    for s in (+1, -1):
+        for w in window:
+            axB.axvline(s * w, color=C["win"], lw=0.8, alpha=0.8)
+    gather_title = "(b) daily CCF gather"
+    if recipe["channels"] > 1:
+        gather_title += f" (mean of {recipe['channels']} channels)"
+    axB.set(xlabel="lapse time (s)", ylabel="day", title=gather_title)
+
+    dvv, valid = golden.recover(d, cfg, uc.eps_max(app))
+    rec = _align(dvv, d["truth"], days, valid)
+    tgt = recipe.get("target")
+    truth_label = f"ground-truth dv/v ({tgt} layer)" if tgt else "ground-truth dv/v"
+    if "truth_other" in d:
+        other = "deep" if tgt == "shallow" else "shallow"
+        axC.plot(days, d["truth_other"] * PCT, color="#9aa4b2", lw=1.2, ls="--",
+                 label=f"other layer ({other}, not targeted)")
+    axC.plot(days, d["truth"] * PCT, color=C["truth"], lw=1.8, label=truth_label)
+    axC.plot(days, rec * PCT, ".", ms=2.6, color=C["rec"],
+             label="recovered (target band)")
+    axC.axhline(0, color="#aaa", lw=0.6)
+    rms = golden._rms(dvv, d["truth"], days, valid)
+    axC.set(xlabel="day", ylabel="dv/v (%)",
+            title=f"(c) dv/v: ground truth vs recovered   |   aligned RMS = {rms*PCT:.4f}%")
+    axC.legend(fontsize=8, frameon=False, loc="best")
+    axC.margins(x=0)
+
+    grade = recipe["grade"]
+    fig.suptitle(
+        f"{case_id}   [{grade} / {recipe['motif']}]   "
+        f"channels={recipe['channels']}, SNR={recipe['snr']}, "
+        f"band={tuple(band)} Hz, window={tuple(window)} s, fs={fs:g} Hz",
+        fontsize=10, y=0.99, color=GRADE_C[grade])
+    outdir.mkdir(parents=True, exist_ok=True)
+    out = outdir / f"{case_id}.png"
+    fig.savefig(out, dpi=130, bbox_inches="tight")
+    plt.close(fig)
+    return out
+
+
+def main(argv: list[str] | None = None) -> int:
+    import argparse
+
+    ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
+    ap.add_argument("--outdir", type=Path, default=OUTDIR,
+                    help="directory for the PNGs (default: tests/data/golden/figs)")
+    args = ap.parse_args(argv)
+    print(f"Plotting {len(golden.CASES)} golden cases -> {args.outdir}")
+    for c in golden.CASES:
+        p = plot_case(c["id"], outdir=args.outdir)
+        print(f"  wrote {p.name}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

src/codameter/frugalmind.py

@@ -41,7 +41,6 @@
 
 from . import golden
 from . import use_cases as uc
-from .deviations import run_pipeline
 
 DATASET_ID = "dvv_processing"
 VERSION = "v0.1"
@@ -138,6 +137,11 @@ def _thresholds(case_id: str) -> dict:
     tol = float(entry["rms_rel_tol"])
     good = max(target * (1.0 + tol), target + 5e-5)
     bad = max(target * 6.0, 3.0e-3)
+    # Depth-targeted cases: anchor "clearly wrong" to the wrong-layer error, so
+    # choosing a band that recovers the other depth scores near zero.
+    wrong = entry["expected"].get("rms_wrong_layer")
+    if wrong and np.isfinite(wrong) and wrong > good:
+        bad = max(good * 1.5, float(wrong))
     return {"rms_target": target, "rms_ceiling": good, "rms_bad": bad}
 
 
@@ -187,7 +191,7 @@ def build_rows(task: str, *, split: str | None = None,
             "scorer_spec": {"name": scorer_name, "config": {}},
             "metadata": {
                 "case_id": case["id"], "use_case": case["use_case"],
-                "kind": case["kind"],
+                "grade": case["grade"],
                 "recommended_config": golden._jsonable(uc.recommend(case["use_case"])),
             },
         })
@@ -284,8 +288,7 @@ def score_param_recommendation(output_text: str, gold: dict) -> float:
     cfg["band"], cfg["window"] = band, window
     d = golden.generate(gold["case_id"])
     try:
-        dvv, valid = run_pipeline(d["ccfs"], d["t"], d["fs"], cfg,
-                                  eps_max=uc.eps_max(gold["use_case"]))
+        dvv, valid = golden.recover(d, cfg, uc.eps_max(gold["use_case"]))
     except Exception:
         return 0.0
     rms = golden._rms(dvv, d["truth"], d["days"], valid)