feat(autodata): v2.2 — Evalchemy-style val-set objective for the meta-loop

ainode/training/autodata/V2_DESIGN.md

@@ -45,11 +45,56 @@ Expose what they ablated, with their best-performers as defaults; the meta-opt s
   synthetically (your small-dataset → big-dataset insight). Real-world seed → synthetic scale.
 
 ## Build slices (each its own /goal, smallest-first)
-- **v2.1 — meta-optimizer** (the yield fix): the `P_t→P_{t+1}` loop + val-set objective. Biggest payoff.
-- **v2.2 — Evalchemy integration**: wire Evalchemy as the verifier/objective (served-endpoint eval).
+- **v2.1 — meta-optimizer** (the yield fix): the `P_t→P_{t+1}` loop + Δ=1-yield proxy. SHIPPED (`meta.py`).
+- **v2.2 — Evalchemy-style val-set objective** (the *real* yield fix): swap the noisy keep-rate proxy
+  for a held-out, GT-graded LIFT objective. **IMPLEMENTED** (`valset.py` + `meta.py objective="valset"`). See below.
 - **v2.3 — recipe knobs + OpenThoughts seeds**: the methodology menu + dataset bootstrapping.
 - **v2.4 — lift ingestion**: documents → structured seeds → synthetic extension.
 
+## v2.2 — the val-set objective (IMPLEMENTED)
+
+**Why (the honest finding, 2026-06-28):** live yield was noisy/low (best ~25%). Root cause is
+the **eval signal**, not the optimizer. v2.1's reward is the per-batch Δ=1 keep-rate — a
+self-graded proxy that (a) re-samples fresh tasks every round (the number wobbles for reasons
+unrelated to P) and (b) rode exact-match arithmetic, whose thin ZPD mislabels correct-but-
+reformatted answers. Optimizing that proxy chases noise.
+
+**What v2.2 optimizes instead — measured lift on a held-out val set:**
+
+    lift(P) = acc(weak | few-shot of D(P))  −  acc(weak alone)      # on a FIXED, GT-labeled val set
+
+- `D(P)` = the round's kept Δ=1 "teacher" traces (strong-solves / weak-fails).
+- The weak solver is scored on a fixed, ground-truth-labeled val set (`val_set`), **cold**
+  (baseline, computed once) and **primed** with a few-shot sample (`val_shots`) of `D(P)`.
+- Both are graded by the **same trusted verifier** the Δ-filter uses (`core.judge_correct` →
+  `verify.py` value-verify + rubric fallback) — the Evalchemy "proven verifier" role.
+- This is **in-context learning as a torch-free proxy for fine-tuning lift**: if the kept
+  traces teach, showing a few raises verified val accuracy; if the batch is junk, lift ≈ 0.
+  Because the val set is fixed and GT-graded, the reward is **stable and comparable
+  round-over-round** — the property raw keep-rate lacks. Evalchemy grades served vLLM OpenAI
+  endpoints, so this drops onto AInode-served models with no new infra (pure HTTP, stdlib-only).
+
+**Where it lives:** `valset.py` (`evaluate`, `valset_lift`, self-check `demo()`); the meta-loop
+(`meta.py meta_optimize(..., objective="valset")`) tracks `best_lift`/`best_score`, feeds the
+lift trajectory to the prompt-optimizer, and stops at `val_target`. The v2.1 `objective="yield"`
+path is the untouched default — nothing regresses. Config keys: `objective`, `val_set`,
+`val_shots`, `val_target` (see `core.AutoDataConfig`). CLI: `--meta --objective valset`.
+
+**Offline-testable:** `valset.demo()`, `meta.demo_valset()`, and pytest cases in
+`tests/test_autodata.py` run the whole objective with injected fake clients — no network. Live
+endpoints are exercised by the operator via the recipe below.
+
+## RESULTS (live fleet run — to be recorded by the operator)
+
+> Placeholder for the first live `--objective valset` run on AInode-served models. Fill in:
+>
+> | date | domain | weak / strong / judge | val_set size | baseline acc | best lift | best round P (excerpt) | rounds | notes |
+> |------|--------|-----------------------|--------------|--------------|-----------|------------------------|--------|-------|
+> | _TBD_ | _math_ | _\<weak\> / \<strong\> / \<judge\>_ | _n_ | _%_ | _Δ_ | _…_ | _k_ | _vs. v2.1 yield-proxy baseline_ |
+>
+> Compare against the v2.1 yield-proxy baseline (best ~25% keep-rate) to confirm the objective
+> swap raises measured teaching value, not just the proxy number.
+
 ## Non-goals (still)
 - Re-deriving curation research (we adopt Open Thoughts' findings).
 - A new eval harness (use Evalchemy).

ainode/training/autodata/__init__.py

@@ -6,3 +6,4 @@
 WEAK solver gets wrong (Δ = I_strong - I_weak == 1).
 """
 from .core import run, AutoDataConfig  # noqa: F401
+from .valset import valset_lift, evaluate  # noqa: F401  # v2.2 Evalchemy-style val-set objective

ainode/training/autodata/core.py

@@ -41,6 +41,22 @@ class AutoDataConfig:
     concurrency: int = 8
     retries: int = 2               # transient HTTP/parse retries per model call
     out: str = ""                  # JSONL output path (optional)
+    # v2.2 — Evalchemy-style val-set objective (see valset.py). Ignored unless
+    # objective == "valset" in the meta-loop; the v2.1 Δ=1-yield proxy stays the default.
+    objective: str = "yield"       # meta-loop reward: "yield" (v2.1 proxy) | "valset" (v2.2 lift)
+    val_set: list = None           # held-out labeled probes [{"input","reference"}] for the objective
+    val_shots: int = 3             # few-shot examples (from kept) used to prime the weak solver on val
+    val_target: float = 0.10       # meta stop-threshold for the valset objective (absolute lift)
+    # v2.2 significance guard — a raw lift on a small val set is mostly sampling noise, so the
+    # meta-loop only ACCEPTS (early-stops) a round when the lift also clears a paired McNemar
+    # test at `alpha` AND the val set has at least `val_min_n` probes. Below that, the objective
+    # is reported but never treated as a genuine round-over-round win.
+    alpha: float = 0.05            # significance level for the paired (McNemar) lift test
+    val_min_n: int = 12            # minimum val-set size before a lift can be accepted as real
+
+    def __post_init__(self):
+        if self.val_set is None:
+            self.val_set = []
 
     @staticmethod
     def from_dict(d: dict) -> "AutoDataConfig":
@@ -51,6 +67,9 @@ def from_dict(d: dict) -> "AutoDataConfig":
             n_tasks=int(d.get("n_tasks", 50)), system_prompt=d.get("system_prompt", ""),
             judge_mode=d.get("judge_mode", "rubric"), concurrency=int(d.get("concurrency", 8)),
             retries=int(d.get("retries", 2)), out=d.get("out", ""),
+            objective=d.get("objective", "yield"), val_set=list(d.get("val_set") or []),
+            val_shots=int(d.get("val_shots", 3)), val_target=float(d.get("val_target", 0.10)),
+            alpha=float(d.get("alpha", 0.05)), val_min_n=int(d.get("val_min_n", 12)),
         )
 
 

ainode/training/autodata/meta.py

@@ -1,12 +1,20 @@
-"""AutoData v2.1 — meta-optimizer.
+"""AutoData v2.1/v2.2 — meta-optimizer.
 
 Close the loop: run v1's Δ-filter batch, read the bucket stats, and have an LLM rewrite
 the generation prompt P_t → P_t+1 toward the zone of proximal development (too_easy ⇒
-harder, too_hard ⇒ easier). Repeat until target yield or max_rounds. Optimizes on the
-Δ=1-yield PROXY only — the val-set objective (Evalchemy) is v2.2.
+harder, too_hard ⇒ easier). Repeat until the target objective or max_rounds.
+
+Two reward objectives (`cfg.objective`, or the `objective=` arg):
+- "yield"  (v2.1, default): the per-batch Δ=1 keep-rate PROXY. Cheap, but noisy/self-graded.
+- "valset" (v2.2): the Evalchemy-style held-out-lift objective (see valset.py). The reward is
+  the weak solver's verified-accuracy LIFT on a fixed labeled val set when primed with a
+  few-shot sample of the round's kept traces — a stable, GT-graded signal that fixes the noisy
+  proxy that motivated v2.2. The loop only *accepts* (early-stops on) a round whose lift clears
+  the target AND passes a paired McNemar significance guard (p ≤ cfg.alpha, n ≥ cfg.val_min_n),
+  so a noisy flip on a small val set can't declare victory. The v2.1 path is untouched.
 
 Reuses v1 (run / call_model / _retry). `core.call_model` is referenced dynamically so the
-self-check's fake injection is seen here too. See demo() at the bottom.
+self-check's fake injection is seen here too. See demo() / demo_valset() at the bottom.
 """
 from __future__ import annotations
 
@@ -16,26 +24,49 @@
 
 from . import core
 from .core import AutoDataConfig, run
+from .valset import valset_items, valset_lift, evaluate
 
 
-def _optimize_prompt(ep, rounds: list, task_spec: str, retries: int = 2) -> str:
+def _optimize_prompt(ep, rounds: list, task_spec: str, retries: int = 2,
+                     objective: str = "yield") -> str:
     """Propose the NEXT generator prompt from the FULL round trajectory, so the optimizer
     learns from an overshoot (too_hard) instead of repeating it. too_easy = both solved
-    (push harder); too_hard = both failed (dial back)."""
-    traj = "\n".join(
-        f"Round {e['round']}: yield={e['yield_pct']}% too_easy={e['too_easy']} "
-        f"too_hard={e['too_hard']}  | prompt: {e['prompt']}"
-        for e in rounds)
+    (push harder); too_hard = both failed (dial back). The trajectory is framed around the
+    active objective (yield proxy vs. held-out val-set lift)."""
+    if objective == "valset":
+        traj = "\n".join(
+            f"Round {e['round']}: lift={e.get('lift')} "
+            f"(val_acc {e.get('val_acc_baseline')}→{e.get('val_acc_primed')}, "
+            f"p={e.get('p_value')} significant={e.get('significant')}) "
+            f"kept={e['kept']}  | prompt: {e['prompt']}"
+            for e in rounds)
+        goal = (
+            "We keep only tasks a STRONG model solves but a WEAK model FAILS (the zone of "
+            "proximal development), then measure LIFT = the weak model's verified-accuracy "
+            "gain on a held-out validation set when it is shown a few of your generated "
+            "examples. Higher lift ⇒ your tasks teach. Trajectory so far:\n"
+            f"{traj}\n\n"
+            f"Task domain: {task_spec}\n"
+            "Propose the NEXT generator prompt to MAXIMIZE lift — LEARN from the trajectory: "
+            "keep what raised lift; if lift stalled or fell, shift the difficulty/coverage so "
+            "the kept traces better cover what the weak model still gets wrong on the val set. ")
+    else:
+        traj = "\n".join(
+            f"Round {e['round']}: yield={e['yield_pct']}% too_easy={e['too_easy']} "
+            f"too_hard={e['too_hard']}  | prompt: {e['prompt']}"
+            for e in rounds)
+        goal = (
+            "We keep only tasks a STRONG model solves but a WEAK model FAILS (the zone of "
+            "proximal development). too_easy = both solved (too easy); too_hard = both failed "
+            "(too hard). Trajectory so far:\n"
+            f"{traj}\n\n"
+            f"Task domain: {task_spec}\n"
+            "Propose the NEXT generator prompt to MAXIMIZE yield — LEARN from the trajectory: "
+            "if a prompt overshot to too_hard, dial difficulty back; if too_easy, push harder; "
+            "converge toward the middle. ")
     msg = (
         "You tune the SYSTEM PROMPT of a task-generator in a synthetic-data pipeline. "
-        "We keep only tasks a STRONG model solves but a WEAK model FAILS (the zone of "
-        "proximal development). too_easy = both solved (too easy); too_hard = both failed "
-        "(too hard). Trajectory so far:\n"
-        f"{traj}\n\n"
-        f"Task domain: {task_spec}\n"
-        "Propose the NEXT generator prompt to MAXIMIZE yield — LEARN from the trajectory: "
-        "if a prompt overshot to too_hard, dial difficulty back; if too_easy, push harder; "
-        "converge toward the middle. "
+        + goal +
         'Return STRICT JSON only: {"prompt": "<the next generator system prompt>"}.'
     )
 
@@ -56,19 +87,42 @@ def _key(ex: dict) -> str:
 
 
 def meta_optimize(config, target_yield: float = 30, max_rounds: int = 4,
-                  optimizer=None, on_round=None) -> dict:
-    """Iteratively rewrite P to raise Δ=1 yield.
+                  optimizer=None, on_round=None, objective=None, target=None) -> dict:
+    """Iteratively rewrite P to raise the reward objective.
+
+    objective: "yield" (v2.1 Δ=1 keep-rate proxy) or "valset" (v2.2 Evalchemy held-out lift).
+    Defaults to cfg.objective. `target` overrides the stop-threshold (yield%% for "yield",
+    absolute lift for "valset"; falls back to target_yield / cfg.val_target).
 
-    Returns {best_prompt, best_yield, dataset (merged+deduped Δ=1), rounds[], out}.
+    Returns {best_prompt, best_yield, best_score, best_lift, best_significant,
+    dataset (merged+deduped Δ=1), rounds[], objective, out}. best_yield stays the best round's
+    Δ=1 yield%% for both objectives (so existing callers keep working); best_score is the
+    active-objective value; best_significant flags whether the winning valset round cleared the
+    paired significance guard (always False for "yield").
     """
     cfg = config if isinstance(config, AutoDataConfig) else AutoDataConfig.from_dict(config)
+    objective = objective or getattr(cfg, "objective", "yield") or "yield"
+    is_valset = objective == "valset"
     opt_ep = optimizer or cfg.challenger          # reuse the challenger endpoint for the rewrite
     out_path = cfg.out
     cfg = copy.copy(cfg)
     cfg.out = ""                                    # we write the merged dataset ourselves
 
+    val_items, val_baseline = [], None
+    if is_valset:
+        val_items = valset_items(cfg)
+        if not val_items:
+            raise ValueError("objective='valset' requires a non-empty val_set in the config")
+        # weak-alone accuracy on the fixed val set — computed once, reused as the lift baseline.
+        # Keep the FULL evaluate() dict (its per_item vector) so each round can run the paired
+        # McNemar significance test against the same fixed baseline.
+        val_baseline = evaluate(cfg, cfg.weak, val_items)
+
+    target_score = target if target is not None else (
+        cfg.val_target if is_valset else target_yield)
+
     rounds, dataset, seen = [], [], set()
-    best = {"yield": -1.0, "prompt": cfg.gen_prompt}
+    best = {"score": float("-inf"), "yield": -1.0, "lift": None, "prompt": cfg.gen_prompt}
     for r in range(1, max_rounds + 1):
         result = run(cfg)
         rep = result["report"]
@@ -80,16 +134,32 @@ def meta_optimize(config, target_yield: float = 30, max_rounds: int = 4,
         entry = {"round": r, "yield_pct": rep["yield_pct"], "kept": rep["kept"],
                  "too_easy": rep["too_easy"], "too_hard": rep["too_hard"],
                  "total": rep["total"], "prompt": cfg.gen_prompt}
+        if is_valset:
+            # attribute the lift to THIS round's kept traces (few-shot from result["kept"])
+            lr = valset_lift(cfg, result["kept"], valset=val_items, baseline=val_baseline)
+            entry.update(lift=lr["lift"], val_acc_primed=lr["acc_primed"],
+                         val_acc_baseline=lr["acc_baseline"], score=lr["lift"],
+                         p_value=lr["p_value"], improved=lr["improved"],
+                         regressed=lr["regressed"], significant=lr["significant"])
+            score = lr["lift"]
+        else:
+            entry["score"] = rep["yield_pct"]
+            score = rep["yield_pct"]
         rounds.append(entry)
-        if rep["yield_pct"] > best["yield"]:
-            best = {"yield": rep["yield_pct"], "prompt": cfg.gen_prompt}
+        if score > best["score"]:
+            best = {"score": score, "yield": rep["yield_pct"], "lift": entry.get("lift"),
+                    "prompt": cfg.gen_prompt, "significant": entry.get("significant", False)}
         if on_round:
             on_round(entry)
-        if rep["yield_pct"] >= target_yield or r == max_rounds:
+        # Accept (early-stop) only on a REAL win. For "valset" a raw lift clearing the target is
+        # not enough — it must also pass the paired significance guard (see valset.valset_lift),
+        # so a noisy flip on a tiny val set can't declare victory. "yield" is unchanged.
+        accepted = score >= target_score and (entry.get("significant", True) if is_valset else True)
+        if accepted or r == max_rounds:
             break
         try:
             # feed the full trajectory so the optimizer corrects an overshoot, not repeats it
-            cfg.gen_prompt = _optimize_prompt(opt_ep, rounds, cfg.task_spec, cfg.retries)
+            cfg.gen_prompt = _optimize_prompt(opt_ep, rounds, cfg.task_spec, cfg.retries, objective)
         except Exception:
             break                                   # can't improve P → stop
 
@@ -98,7 +168,9 @@ def meta_optimize(config, target_yield: float = 30, max_rounds: int = 4,
             for ex in dataset:
                 f.write(json.dumps(ex) + "\n")
     return {"best_prompt": best["prompt"], "best_yield": best["yield"],
-            "dataset": dataset, "rounds": rounds, "out": out_path}
+            "best_score": best["score"], "best_lift": best["lift"],
+            "best_significant": best.get("significant", False),
+            "dataset": dataset, "rounds": rounds, "objective": objective, "out": out_path}
 
 
 def demo() -> None:
@@ -138,5 +210,56 @@ def fake(ep, messages, json_mode):
         core.call_model = core._http_chat
 
 
+def demo_valset() -> None:
+    """Self-check for the v2.2 val-set objective: a fake world where a higher TEACH level in
+    the generator prompt makes the kept strong-traces carry a stronger HINT, which in turn
+    lets the primed weak solver clear more held-out val probes. The meta-loop must optimize
+    the held-out LIFT (not raw yield) and converge to the higher-TEACH prompt."""
+    def fake(ep, messages, json_mode):
+        sysmsg = messages[0]["content"] if messages and messages[0]["role"] == "system" else ""
+        last = messages[-1]["content"]
+        if json_mode and "task-generator" in last:              # optimizer: bump TEACH
+            ts = [int(x) for x in re.findall(r"TEACH=(\d+)", last)]
+            return json.dumps({"prompt": f"TEACH={(max(ts) if ts else 0) + 1}"})
+        if json_mode and "Generate" in last:                    # challenger: emit tasks at TEACH
+            t = int((re.search(r"TEACH=(\d+)", sysmsg) or [0, "0"])[1])
+            return json.dumps({"tasks": [{"input": f"K{t}_{i}", "reference": "1"} for i in range(4)]})
+        # solver on a GENERATED task "K{t}_i": strong emits a good trace carrying HINT=t
+        # (verifies to 1); weak fails (verifies to 0) -> Δ=1 kept.
+        m = re.match(r"K(\d+)_", last)
+        if m:
+            t = int(m.group(1))
+            return f"HINT={t}, answer = 1" if ep.model == "strong" else "answer = 0"
+        # val probe "L{req}": the primed weak solver clears it iff a HINT>=req is in context
+        if last and last[0] == "L" and last[1:].isdigit():
+            req = int(last[1:])
+            hints = [int(tok.split("=", 1)[1].rstrip(","))
+                     for mm in messages for tok in str(mm.get("content", "")).split()
+                     if tok.startswith("HINT=") and tok.split("=", 1)[1].rstrip(",").isdigit()]
+            return "answer = 1" if (hints and max(hints) >= req) else "answer = 0"
+        return "answer = 0"
+
+    core.call_model = fake
+    try:
+        cfg = AutoDataConfig(
+            task_spec="x", gen_prompt="TEACH=0", n_tasks=4, concurrency=1, judge_mode="verify",
+            objective="valset", val_shots=4, val_target=0.9,
+            challenger=core.Endpoint("u", "challenger"), weak=core.Endpoint("u", "weak"),
+            strong=core.Endpoint("u", "strong"), judge=core.Endpoint("u", "judge"),
+            val_set=[{"input": "L1", "reference": "1"}, {"input": "L2", "reference": "1"},
+                     {"input": "L3", "reference": "1"}])
+        out = meta_optimize(cfg, max_rounds=5)
+        rounds = out["rounds"]
+        assert out["objective"] == "valset", out
+        assert rounds[0]["lift"] == 0.0, rounds            # TEACH=0 -> HINT=0 clears no probe
+        assert rounds[-1]["lift"] > rounds[0]["lift"], rounds   # lift rose as TEACH climbed
+        assert out["best_lift"] > 0 and out["best_score"] == out["best_lift"], out
+        assert "TEACH=" in out["best_prompt"], out
+        print("autodata meta valset demo OK:", [(r["round"], r["lift"]) for r in rounds])
+    finally:
+        core.call_model = core._http_chat
+
+
 if __name__ == "__main__":
     demo()
+    demo_valset()