Graded 30-case golden benchmark (easy/medium/hard, multi-channel hard)#14
Open
mdenolle wants to merge 2 commits into
Open
Graded 30-case golden benchmark (easy/medium/hard, multi-channel hard)#14mdenolle wants to merge 2 commits into
mdenolle wants to merge 2 commits into
Conversation
Replace the 10 mainstream/edge golden cases with a graded benchmark: three difficulty grades, ten cases each, spanning the applications. - easy (validation): pure seasonal signal, high SNR (8-12), single channel. Best-practice recovery is well under 0.2 % RMS. - medium (validation): a transient coseismic-style drop with logarithmic partial healing, plus more measurement noise (SNR 3-5). - hard (test): a MULTI-CHANNEL (4-channel) problem whose truth combines a transient drop-and-heal, a full hydrological seasonal cycle, and a long-term trend, at low SNR (2-4) with waveform decorrelation. Channels are measured independently and aggregated to a network dv/v. golden.py rewrite: - motif builders (seasonal / transient / composite) with per-application amplitudes and timescales; grades cycle through the 6 applications. - golden.recover(d, cfg, eps): single scoring entry point handling single- and multi-channel cases (per-channel run_pipeline, then average the per-channel dv/v). frugalmind and the manifest oracle both go through it. - generate() emits `channels` (3D) for multi-channel cases and downcasts the large CCF arrays to float32 in the cache. representative_case()/ MAINSTREAM_BY_USE_CASE kept for the advisor and sweep. - manifest v2: grade/motif/channels/snr/decorr per case. Verified: the benchmark discriminates on the choices that matter (a moving reference scores 0.0 on a strong-trend hard case, ~0.19 on a drop case), while being band-insensitive for a homogeneous medium (documented). frugalmind rows now 30 per suite; tests updated to grade ids. Full suite: 214 passed, 1 skipped. Follow-ups (separate branches): bench (#13) must switch score_cell to golden.recover and drop the old ids; the live-site golden gallery regenerates for the 30 cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR replaces the prior 10-case “mainstream/edge” golden set with a graded 30-case benchmark (easy/medium/hard), introduces multi-channel “hard” cases, and routes scoring through a single golden.recover() entry point to support correct aggregation.
Changes:
- Reworked
codameter.goldento generate 30 graded cases (including 4-channel hard cases) and addedgolden.recover()for unified single-/multi-channel scoring. - Updated the committed golden manifest to v2 (grade/motif/channels/etc.) and refreshed regression + FrugalMind export tests to use the new case IDs/fields.
- Added a plotting script (
scripts/plot_golden.py) and updated advisor reference docs + gitignore for generated figures.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
src/codameter/golden.py |
Implements graded case construction, multi-channel synthesis, recover() aggregation, cache format updates, and manifest v2 generation. |
tests/data/golden/manifest.json |
Updates committed oracle to v2 and expands to 30 graded cases with per-case metadata + expected RMS. |
tests/test_golden.py |
Updates regression tests for graded cases, manifest v2, and validates recover() on single vs multi-channel cases. |
src/codameter/frugalmind.py |
Switches scoring to golden.recover() and updates metadata from kind to grade. |
tests/test_frugalmind_export.py |
Updates tests to reference new graded case IDs. |
scripts/plot_golden.py |
Adds a deterministic visualization tool for all graded cases (including aggregated multi-channel recovery). |
tests/data/golden/.gitignore |
Ignores generated verification figures directory. |
.claude/skills/codameter-advisor/references/validation_loop.md |
Updates advisor documentation to reference graded cases and new IDs. |
.claude/skills/codameter-advisor/references/golden_datasets.md |
Updates advisor documentation to describe the graded benchmark and recover() aggregation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
243
to
249
| if cache and cache_file.exists(): | ||
| z = np.load(cache_file, allow_pickle=False) | ||
| d = {k: z[k] for k in z.files} | ||
| d["fs"] = float(d["fs"]) | ||
| d["use_case"] = str(d["use_case"]) | ||
| d["grade"] = str(d["grade"]) | ||
| return d |
Comment on lines
+150
to
153
| # Back-compat alias: application -> a representative (easy) case. | ||
| MAINSTREAM_BY_USE_CASE = { | ||
| c["use_case"]: c["id"] for c in CASES if c["kind"] == "mainstream" | ||
| app: representative_case(app) for app in {c["use_case"] for c in CASES} | ||
| } |
On top of multi-channel + composite, the hard grade is now a depth-dependent medium: a shallow (high-frequency) layer carries the coseismic drop-and-heal plus the full hydrological seasonal cycle; a deep (low-frequency) layer carries the long-term trend. The two layers live in separated bands, so the measurement band selects the depth. Each hard case targets one depth (alternating), and the recommended config's band must match it. - golden.py: _truth_shallow / _truth_deep / _depth_bands; the hard _build path sums both layers per channel; d["truth"] is the targeted layer (d["truth_other"] kept for plotting). compute_metrics adds rms_wrong_layer (the error from recovering the other depth) as the scoring anchor; manifest records two_layer/target. - frugalmind.py: anchor rms_bad to rms_wrong_layer so a band that recovers the wrong depth scores ~0 (verified: target band 1.0, wrong band 0.0). - plot_golden.py: use the target band and overlay the non-targeted layer. - tests: band-selects-depth (target band beats wrong band > 2x and tracks the target layer) and manifest depth-target checks. Verified: measuring in the target band recovers the target layer; a wrong band recovers the other layer and is ~4-6x worse in RMS, 0.0 in score. Full suite: 216 passed, 1 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the 10 mainstream/edge golden cases with a graded benchmark: three difficulty grades, ten cases each (30 total), spanning the monitoring applications.
The grades
Case ids are
{grade}-{application}-{nn}(e.g.easy-volcano-01,hard-groundwater-08); each grade cycles through the six applications.The hard grade is the multi-channel case you asked for: N cross-component channels with independent noise, measured per channel and aggregated to a network dv/v.
golden.pyrewriteseasonal/transient/composite) with per-application amplitudes and timescales.golden.recover(d, cfg, eps)— one scoring entry point that handles single- and multi-channel cases (per-channelrun_pipeline, then average the per-channel dv/v).frugalmindand the manifest oracle both route through it.generate()emitschannels(3D) for multi-channel cases and downcasts the large CCF arrays to float32 in the (gitignored) cache.representative_case()/MAINSTREAM_BY_USE_CASEkept for the advisor and sweep.Verification
scripts/plot_golden.py) were eyeballed per grade; the hard composite clearly superposes drop + seasonal + trend across 4 noisy channels, aggregated to a clean recovery.movingreference scores 0.0 on a strong-trend hard case and ~0.19 on a drop case, while the recommended config scores 1.0.Follow-ups (separate branches)
score_cellmust switch togolden.recover(to aggregate hard multi-channel cases) and drop the old*_mainstreamids. I'll apply this to that branch.🤖 Generated with Claude Code