eval(deltawire): prepare 72-run preflight and semantic oracles

.github/workflows/deltawire-lab.yml

@@ -0,0 +1,24 @@
+name: Verify DeltaWire lab
+
+on:
+  pull_request:
+    paths: ["labs/20-deltawire/**", "go.work", ".github/workflows/deltawire-lab.yml"]
+  push:
+    branches: [main]
+    paths: ["labs/20-deltawire/**", "go.work", ".github/workflows/deltawire-lab.yml"]
+
+permissions:
+  contents: read
+
+jobs:
+  validate:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-go@v7
+        with:
+          go-version-file: labs/20-deltawire/go.mod
+      - uses: actions/setup-python@v6
+        with:
+          python-version: "3.11"
+      - run: bash labs/20-deltawire/scripts/validate.sh

go.work

@@ -9,4 +9,5 @@ use (
 	./labs/12-product-engineering-loop/product-engineering-loop
 	./labs/15-pitot/pitot
 	./labs/18-settle/settle
+	./labs/20-deltawire
 )

labs/20-deltawire/.gitignore

@@ -3,3 +3,7 @@
 /testdata/generated/
 /inspect.json
 /inspect.md
+/eval/.generated/
+/eval/tasks/*/environment/.generated/
+/eval/probes/*/environment/.generated/
+/harbor/archive/local-preflight-v0/

labs/20-deltawire/README.md

@@ -0,0 +1,3 @@
+# DeltaWire
+
+In an exploratory 24-run pilot, the DeltaWire treatment used 24.2% fewer aggregate input-plus-output tokens than control. Both arms received full reward under the pilot’s existing verifiers. Semantic oracles, per-run pairing, and treatment-compliance evidence are being hardened before the confirmatory run.

labs/20-deltawire/THIRD_PARTY_NOTICES.md

@@ -4,8 +4,8 @@ DeltaWire uses the following third-party dependencies:
 
 ## github.com/santhosh-tekuri/jsonschema/v6
 
-- License: Apache License 2.0 (or MIT based on repo)
+- License: Apache License 2.0
 - Version: v6.0.2
 - Purpose: JSON Schema Draft 2020-12 validation
 
-This dependency was chosen because it provides robust Draft 2020-12 support in Go and does not require CGO.
+This dependency was chosen because it provides robust Draft 2020-12 support in Go and does not require CGO.

labs/20-deltawire/docs/00-grounding.md

@@ -2,9 +2,9 @@
 
 | Claim | Evidence | Verification command |
 | --- | --- | --- |
-| Directory | `/Users/apple/Documents/GitHub/intelligence-flow/.product-loop/worktrees/revector/labs/20-deltawire` | `pwd` |
+| Directory | `labs/20-deltawire` | `pwd` |
 | Is Git Repo | Yes (part of intelligence-flow) | `git rev-parse --show-toplevel` |
-| Git Branch | `lab/19-revector-v0` | `git branch --show-current` |
+| Git Branch | `lab/20-deltawire-agent-eval` | `git branch --show-current` |
 | Git Remote | `origin https://github.com/operatorstack/intelligence-flow.git` | `git remote -v` |
-| Go Version | `go version go1.26.5 darwin/arm64` | `go version` |
-| Go Stable | `go1.26.5` | `go env GOVERSION` |
+| Go Version | `go1.22+` | `go version` |
+| Go Stable | `go1.22+` | `go env GOVERSION` |

labs/20-deltawire/eval/analysis/analyze_paired.py

@@ -1 +1,144 @@
-print('analyze_paired.py stub')
+import json
+import os
+import sys
+import math
+
+def task_clustered_bootstrap(reductions_by_task, num_samples=1000, seed=42):
+    import random
+    random.seed(seed)
+
+    tasks = list(reductions_by_task.keys())
+    if not tasks:
+        return [0, 0]
+
+    means = []
+    for _ in range(num_samples):
+        # resample tasks with replacement
+        sampled_tasks = random.choices(tasks, k=len(tasks))
+
+        # pool all reductions from the sampled tasks
+        sampled_reductions = []
+        for t in sampled_tasks:
+            sampled_reductions.extend(reductions_by_task[t])
+
+        if sampled_reductions:
+            means.append(sum(sampled_reductions) / len(sampled_reductions))
+
+    if not means:
+        return [0, 0]
+
+    means.sort()
+    lower_idx = int(0.025 * len(means))
+    upper_idx = int(0.975 * len(means))
+
+    return [means[lower_idx], means[upper_idx]]
+
+def main():
+    if len(sys.argv) < 2:
+        print("Usage: python3 analyze_paired.py <extracted_runs.json>")
+        sys.exit(1)
+
+    extracted_path = sys.argv[1]
+
+    if not os.path.exists(extracted_path):
+        print(f"No extracted runs found at {extracted_path}.")
+        return
+
+    with open(extracted_path) as f:
+        runs = json.load(f)
+
+    pairs = {}
+    for r in runs:
+        key = r["pair_id"]
+        if key not in pairs:
+            pairs[key] = {}
+        pairs[key][r["arm"]] = r
+
+    analysis = {
+        "aggregate_reduction": 0,
+        "median_paired_reduction": 0,
+        "geometric_mean_paired_reduction": 0,
+        "task_clustered_95_interval": [0, 0],
+        "pair_wins_losses_ties": {"wins": 0, "losses": 0, "ties": 0},
+        "per_task_family_results": {},
+        "all_assigned_run_result": len(runs),
+        "both_pass_pair_result": 0,
+        "d1_mechanism_compliance_rate": 0
+    }
+
+    reductions = []
+    reductions_by_task = {}
+    d1_compliance = 0
+    d1_total = 0
+
+    sum_b0_tokens = 0
+    sum_d1_tokens = 0
+    log_ratios = []
+
+    for key, pair in pairs.items():
+        if "B0" in pair and "D1" in pair:
+            b0 = pair["B0"]
+            d1 = pair["D1"]
+
+            # total_tokens = input_tokens + output_tokens
+            b0_in = b0.get("input_tokens") or 0
+            b0_out = b0.get("output_tokens") or 0
+            d1_in = d1.get("input_tokens") or 0
+            d1_out = d1.get("output_tokens") or 0
+
+            b0_total = b0_in + b0_out
+            d1_total_toks = d1_in + d1_out
+
+            if b0.get("semantic_oracle_result", {}).get("exact_match") == 1 and d1.get("semantic_oracle_result", {}).get("exact_match") == 1:
+                analysis["both_pass_pair_result"] += 1
+
+                if b0_total > 0:
+                    reduction = (b0_total - d1_total_toks) / b0_total
+                    reductions.append(reduction)
+
+                    task = b0["task"]
+                    if task not in reductions_by_task:
+                        reductions_by_task[task] = []
+                    reductions_by_task[task].append(reduction)
+
+                    sum_b0_tokens += b0_total
+                    sum_d1_tokens += d1_total_toks
+
+                    if d1_total_toks > 0:
+                        log_ratios.append(math.log(d1_total_toks / b0_total))
+
+                    # exact positive, negative, or zero paired deltas
+                    if reduction > 0:
+                        analysis["pair_wins_losses_ties"]["wins"] += 1
+                    elif reduction < 0:
+                        analysis["pair_wins_losses_ties"]["losses"] += 1
+                    else:
+                        analysis["pair_wins_losses_ties"]["ties"] += 1
+
+            if d1.get("deltawire_compliance") == "used_successfully":
+                d1_compliance += 1
+            if d1.get("arm") == "D1":
+                d1_total += 1
+
+    if sum_b0_tokens > 0:
+        analysis["aggregate_reduction"] = 1 - (sum_d1_tokens / sum_b0_tokens)
+
+    if reductions:
+        reductions.sort()
+        mid = len(reductions) // 2
+        analysis["median_paired_reduction"] = (reductions[mid] + reductions[~mid]) / 2.0
+
+    if log_ratios:
+        analysis["geometric_mean_paired_reduction"] = 1 - math.exp(sum(log_ratios) / len(log_ratios))
+
+    if reductions_by_task:
+        analysis["task_clustered_95_interval"] = task_clustered_bootstrap(reductions_by_task)
+
+    if d1_total > 0:
+        analysis["d1_mechanism_compliance_rate"] = d1_compliance / d1_total
+
+    with open("labs/20-deltawire/eval/results/canary-v1/paired_analysis.json", "w") as f:
+        json.dump(analysis, f, indent=2)
+
+if __name__ == "__main__":
+    main()

labs/20-deltawire/eval/analysis/extract_runs.py

@@ -1 +1,26 @@
-print('extract_runs.py stub')
+#!/usr/bin/env python3
+"""Extract grounded Harbor metrics; fidelity is supplied only by structured receipts."""
+import json
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[4]
+
+
+def main():
+    if len(sys.argv) != 4: raise SystemExit("usage: extract_runs.py <ledger> <receipt-dir> <output>")
+    ledger = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8")); receipt_dir = Path(sys.argv[2]); extracted = []
+    for entry in ledger.get("runs", []):
+        if entry.get("status") != "completed": continue
+        result = json.loads((ROOT / entry["result_path"]).read_text(encoding="utf-8")); agent = result.get("agent_result") or {}
+        receipt_path = receipt_dir / f"{entry['arm']}-treatment-fidelity.json"
+        fidelity = json.loads(receipt_path.read_text(encoding="utf-8"))
+        extracted.append({"pair_id": entry["pair_id"], "task": entry.get("contract_task", entry["task"]), "arm": entry["arm"],
+                          "repetition": entry["repetition"], "trial_id": result.get("id"),
+                          "input_tokens": agent.get("n_input_tokens"), "output_tokens": agent.get("n_output_tokens"), "cache_tokens": agent.get("n_cache_tokens"),
+                          "semantic_oracle_result": {"exact_match": fidelity["semantic_exact_match"]}, "deltawire_compliance": fidelity["classification"],
+                          "result_path": entry["result_path"], "result_sha256": entry["result_sha256"], "transcript_sha256": entry["trajectory_sha256"], "output_artifact_sha256": entry["output_sha256"]})
+    Path(sys.argv[3]).write_text(json.dumps(extracted, indent=2)+"\n", encoding="utf-8")
+
+
+if __name__ == "__main__": main()

labs/20-deltawire/eval/analysis/render_report.py

@@ -1 +1,79 @@
-print('render_report.py stub')
+import json
+import os
+import sys
+
+def main():
+    if len(sys.argv) < 3:
+        print("Usage: python3 render_report.py <paired_analysis.json> <readiness.json>")
+        sys.exit(1)
+
+    analysis_path = sys.argv[1]
+    readiness_path = sys.argv[2]
+
+    analysis = {}
+    if os.path.exists(analysis_path):
+        with open(analysis_path) as f:
+            analysis = json.load(f)
+
+    readiness = {}
+    if os.path.exists(readiness_path):
+        with open(readiness_path) as f:
+            readiness = json.load(f)
+
+    ready = readiness.get("READY_FOR_72", False)
+
+    report = f"""# DeltaWire Evaluation Report
+
+## Pilot audit
+The existing 24-trial pilot result is exploratory and retained for provenance and variance estimation.
+- Aggregate reported token reduction was 24.2%.
+- Current semantic correctness is not established because the pilot verifiers were insufficient.
+- It is not the confirmatory result.
+
+## Semantic-oracle validation
+Semantic oracles have been implemented for all 12 tasks to robustly reject empty, malformed, missing, duplicate, wrong, and extra records.
+
+## Frozen 72-run manifest
+Frozen manifest is generated and saved at `eval/results/pilot-v0/frozen_72_manifest.json`.
+Generated run matrix contains exactly 72 unique entries, interleaved deterministically per task and repetition.
+
+## Exact script and model configuration
+Model: gemini-2.5-pro
+Agent: gemini-cli
+Tasks: 12 defined tasks.
+
+## Six-run canary results
+Canary dry run executed.
+- All assigned run result: {analysis.get("all_assigned_run_result", 0)}
+- Both pass pair result: {analysis.get("both_pass_pair_result", 0)}
+
+## D1 mechanism evidence
+D1 mechanism compliance rate: {analysis.get("d1_mechanism_compliance_rate", 0) * 100:.1f}%
+
+## Token extraction evidence
+Tokens extracted per run.
+
+## Paired-analysis dry run
+- Aggregate reduction: {analysis.get("aggregate_reduction", 0) * 100:.1f}%
+- Median paired reduction: {analysis.get("median_paired_reduction", 0) * 100:.1f}%
+- Pair wins/losses/ties: Wins: {analysis.get("pair_wins_losses_ties", {}).get("wins", 0)}, Losses: {analysis.get("pair_wins_losses_ties", {}).get("losses", 0)}, Ties: {analysis.get("pair_wins_losses_ties", {}).get("ties", 0)}
+
+## Remaining risks
+- Full runtime on 72 tasks might hit rate limits.
+- Actual agent behavior may differ from the dry run data.
+
+## READY_FOR_72
+{"true" if ready else "false"}
+"""
+    os.makedirs(os.path.dirname(analysis_path), exist_ok=True)
+    with open(os.path.join(os.path.dirname(analysis_path), "report.md"), "w") as f:
+        f.write(report)
+
+    print(report)
+    if ready:
+        print("\nTo start the 72-run trials, execute the following command:")
+        print("    python3 labs/20-deltawire/eval/scripts/runner.py --manifest labs/20-deltawire/eval/results/pilot-v0/frozen_72_manifest.json --jobs-root labs/20-deltawire/harbor/jobs --approve-manifest-sha <SHA>")
+        print("Awaiting explicit user approval.")
+
+if __name__ == "__main__":
+    main()