SLM-338: unified evidence publisher for plan experiments (AP-037)

README.md

@@ -319,6 +319,18 @@ instead of memorizing paths. Selected immutable snapshots publish to Git with
 Every new run writes `outputs/runs/<id>/trace.json` and OTLP JSONL signals under
 `outputs/traces/<trace-id>/`. Set `OTEL_EXPORTER_OTLP_ENDPOINT` for an optional
 remote OTLP mirror; detailed domain traces remain local and linked by trace ID.
+Set `LANGSMITH_TRACING=true` and provide `LANGSMITH_API_KEY` to export an
+additional, best-effort aggregate trace to the `slm-training` LangSmith project
+(`LANGSMITH_PROJECT`, `LANGSMITH_ENDPOINT`, and `LANGSMITH_WORKSPACE_ID` are
+optional overrides). This exports run/suite metrics, version stamps, gate
+verdicts, and AgentV summaries only—never prompts, targets, completions,
+checkpoints, raw logs, or credentials. Local evidence and AgentEvals remain
+authoritative when LangSmith is unavailable. Install the harness-side client
+with `pip install -e '.[observability]'`; the shared trace boundary loads the
+repository's ignored `.env` without overriding environment variables provided
+by CI or a shell. Standard OTLP settings are honored as well:
+`OTEL_SERVICE_NAME`, `OTEL_RESOURCE_ATTRIBUTES`, signal-specific endpoints,
+and `OTEL_EXPORTER_OTLP_TIMEOUT` (milliseconds; bounded to five seconds).
 
 The flush pipeline remains: curated seeds + RICO + Awwwards → deterministic
 quality synth → per-record DESIGN.md + OpenUI validate → quality gates → stable

docs/design/abstract-planning-ap-037-fixture.json

@@ -0,0 +1,134 @@
+{
+  "agentv": {
+    "calibration_version": "agentv-cal-v3",
+    "n": 24,
+    "path": "outputs/ap037/agentv.jsonl"
+  },
+  "campaign_id": "ap-037-fixture",
+  "campaign_manifest_sha256": "00601c6d1c08dbe7ca22a3402f2c9e93d9178d9634018e0c6db29baef3215f41",
+  "checkpoint": {
+    "name": "ap037_fixture.pt",
+    "sha256": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff",
+    "size_bytes": 1024,
+    "uri": "outputs/runs/ap037_fixture/checkpoints/last.pt"
+  },
+  "claim_class": "diagnostic",
+  "content_sha256": "2e683dd3d647c6e5c7a4c38ad0a2b324348c9a2134708a38fffb6e1d5df8d29c",
+  "controls": [
+    {
+      "description": "Gold plan oracle ceiling.",
+      "kind": "oracle",
+      "metrics": {
+        "binder_reference_f1": 0.9583
+      },
+      "n": 24
+    },
+    {
+      "description": "Random plan negative control.",
+      "kind": "random",
+      "metrics": {
+        "binder_reference_f1": 0.2083
+      },
+      "n": 24
+    },
+    {
+      "description": "Empty plan negative control.",
+      "kind": "empty",
+      "metrics": {
+        "binder_reference_f1": 0.1667
+      },
+      "n": 24
+    },
+    {
+      "description": "Shuffled step-order negative control.",
+      "kind": "shuffled",
+      "metrics": {
+        "binder_reference_f1": 0.2292
+      },
+      "n": 24
+    }
+  ],
+  "data_snapshot_sha256": "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
+  "human_audit": {
+    "calibration_version": "human-cal-v1",
+    "n": 12,
+    "path": "outputs/ap037/human_audit.md"
+  },
+  "interventions": [
+    {
+      "delta": -0.4375,
+      "description": "Shuffle plan step order",
+      "intervention_id": "shuffle-steps",
+      "metric": "binder_reference_f1"
+    }
+  ],
+  "latency": {
+    "generation_seconds": 41.5,
+    "p95_seconds": 58.2,
+    "plan_seconds": 3.25,
+    "total_seconds": 51.5,
+    "verification_seconds": 6.75
+  },
+  "metrics": [
+    {
+      "artifact": {
+        "name": "raw.jsonl",
+        "sha256": "1111111111111111111111111111111111111111111111111111111111111111",
+        "size_bytes": 2048,
+        "uri": "outputs/ap037/raw.jsonl"
+      },
+      "binder_reference_f1": 0.4583,
+      "meaning_v2": 0.3125,
+      "n": 24,
+      "parse_rate": 0.625,
+      "path": "raw"
+    },
+    {
+      "artifact": {
+        "name": "constrained.jsonl",
+        "sha256": "2222222222222222222222222222222222222222222222222222222222222222",
+        "size_bytes": 2048,
+        "uri": "outputs/ap037/constrained.jsonl"
+      },
+      "binder_reference_f1": 0.6667,
+      "meaning_v2": 0.5417,
+      "n": 24,
+      "parse_rate": 1.0,
+      "path": "constrained"
+    },
+    {
+      "artifact": {
+        "name": "repaired.jsonl",
+        "sha256": "3333333333333333333333333333333333333333333333333333333333333333",
+        "size_bytes": 2048,
+        "uri": "outputs/ap037/repaired.jsonl"
+      },
+      "binder_reference_f1": 0.7083,
+      "meaning_v2": 0.5833,
+      "n": 24,
+      "parse_rate": 1.0,
+      "path": "repaired"
+    }
+  ],
+  "schema": "AbstractPlanningResultV1",
+  "source_commit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+  "source_dirty": false,
+  "verifier_gates": [
+    {
+      "gate_id": "meaning-improves",
+      "observed": 0.2084,
+      "passed": true,
+      "threshold": 0.01
+    }
+  ],
+  "version_stamp": {
+    "code_commit": "368dcafbd8841ce11956070695b41fdb0bbd6381",
+    "code_dirty": true,
+    "components": {
+      "harness.core": "v9",
+      "harness.experiments.slm338_evidence_publisher": "v1"
+    },
+    "stamp_schema": "version_stamp/v1",
+    "stamped_at": "2026-07-25T00:59:13.813116+00:00"
+  }
+}

docs/design/abstract-planning-ap-037-fixture.md

@@ -0,0 +1,43 @@
+# Abstract planning result — `ap-037-fixture`
+
+- Schema: `AbstractPlanningResultV1`
+- Claim class: **diagnostic**
+- Campaign manifest sha256: `00601c6d1c08dbe7ca22a3402f2c9e93d9178d9634018e0c6db29baef3215f41`
+- Source commit: `aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa` (dirty: False)
+- Data snapshot sha256: `eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee`
+- Checkpoint: `ap037_fixture.pt` sha256 `ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff`
+
+## Decode-path metrics
+
+| Path | n | meaning-v2 | binder F1 | parse |
+| --- | --- | --- | --- | --- |
+| raw | 24 | 0.3125 | 0.4583 | 0.6250 |
+| constrained | 24 | 0.5417 | 0.6667 | 1.0000 |
+| repaired | 24 | 0.5833 | 0.7083 | 1.0000 |
+
+## Plan controls
+
+- **oracle** (n=24): binder_reference_f1=0.9583
+- **random** (n=24): binder_reference_f1=0.2083
+- **empty** (n=24): binder_reference_f1=0.1667
+- **shuffled** (n=24): binder_reference_f1=0.2292
+
+## Causal interventions
+
+- `shuffle-steps` — Shuffle plan step order (binder_reference_f1 delta -0.4375)
+
+## Latency / compute (seconds)
+
+| plan | generation | verification | total | p95 |
+| --- | --- | --- | --- | --- |
+| 3.2500 | 41.5000 | 6.7500 | 51.5000 | 58.2000 |
+
+## Verifier gates
+
+| Gate | Observed | Threshold | Passed |
+| --- | --- | --- | --- |
+| meaning-improves | 0.2084 | 0.0100 | **True** |
+
+- AgentV: `outputs/ap037/agentv.jsonl` (n=24, calibration `agentv-cal-v3`)
+
+- Human audit: `outputs/ap037/human_audit.md` (n=12, calibration `human-cal-v1`)

docs/design/abstract-planning-evidence-contract.md

@@ -0,0 +1,42 @@
+# Abstract planning evidence contract (AP-037 / SLM-338)
+
+One canonical result object — `AbstractPlanningResultV1`
+(`src/slm_training/harnesses/autoresearch/planning_result.py`) — is the single
+source of truth for every plan-experiment publication. One command,
+`scripts/publish_planning_result.py`, emits the canonical JSON, the narrative
+Markdown (rendered from the JSON only, never hand-edited), and a
+model-card-ready roster row from that one object. No manual metric copying.
+
+## Fail-closed publication
+
+`publication_blockers(result)` must be empty before anything is emitted. It
+blocks on:
+
+- missing / `UNKNOWN` / malformed provenance (`source_commit`,
+  `data_snapshot_sha256`; plus a checkpoint `ArtifactRef` for promotion classes);
+- missing or malformed locked campaign manifest sha256
+  (`campaign_manifest_sha256`, the SLM-337 `campaign_manifest_sha256` digest of
+  the locked `ExperimentCampaignV1`);
+- missing total latency (`latency.total_seconds` must be present and positive);
+- for `promotion_candidate` / `ship_gate` claim classes: all three decode paths
+  (`raw`, `constrained`, `repaired`), at least one negative plan control
+  (`random` / `empty` / `shuffled`), and a checkpoint reference.
+
+`diagnostic` / `wiring` / `fixture` / `screening` results publish without the
+promotion-class requirements but still fail closed on provenance, manifest
+hash, and latency.
+
+## Disposition log
+
+The section below is append-only and keyed by locked campaign manifest hash;
+replaying the same campaign hash is a no-op. Sections are generated by
+`scripts/publish_planning_result.py --append-disposition`; never hand-edit.
+Historical iter documents (`docs/design/iter-*`) stay immutable — the publisher
+only writes its own artifacts.
+
+## Campaign `00601c6d1c08dbe7ca22a3402f2c9e93d9178d9634018e0c6db29baef3215f41`
+
+- Campaign id: `ap-037-fixture`
+- Claim class: **diagnostic**
+- Result content sha256: `2e683dd3d647c6e5c7a4c38ad0a2b324348c9a2134708a38fffb6e1d5df8d29c`
+- Evidence: `docs/design/abstract-planning-ap-037-fixture.json`

docs/design/agentv-evaluation.md

@@ -29,6 +29,10 @@ AgentEvals assertion results has `authority: "AgentEvals assertions"`.
    under `<run-dir>/evals/`.
 5. `write_ship_gates` projects those assertion results into the compatibility
    `gates.json` shape and records the AgentEvals authority.
+6. When opt-in LangSmith tracing is configured, the runner publishes its
+   aggregate AgentV summary as a child of the existing W3C-correlated run
+   trace. This is observational only; AgentEvals assertions, local artifacts,
+   and ship gates retain authority if export fails.
 
 The `agentv` npm package is retained for the canonical CLI and dashboard. In
 the pinned release its published package is CLI-only, so programmatic execution
@@ -82,3 +86,4 @@ The implementation check is recorded in
 | --- | --- | --- | --- |
 | 2026-07-14 | CPU, steps 0, no model backend; AgentV SDK fixture plus model/loss/task/train-loop harness tests | 34/34 focused tests passed; SDK fixture wrote valid AgentEvals JSONL and AgentV artifacts; dependency audit has 0 high/critical findings | Tooling wiring only; no checkpoint, model score, or ship gate was produced |
 | 2026-07-23 | CPU, steps 0, no model backend; focused assertion-authority tests with the pinned SDK | AgentEvals JSONL carried required code-graders over raw criteria; the runner produced 1/1 passing fixture criteria with 0 execution errors; 135 focused gate/consumer checks and all 6 interpreted page validations passed. Dashboard production build remained environment-blocked because the locked `@openfeature/web-sdk` dependency was unavailable and automatic review rejected installation before execution. | Tooling and gate-authority wiring only; no checkpoint or model-quality claim |
+| 2026-07-24 | Local summary-only telemetry configuration smoke; LangSmith `slm-training` project, two-second bounded flush, standard OTLP resource defaults | W3C root trace `fdf78e5360a6647b09fa0882a2c5d4ec` was published and read back as `slm.telemetry.config` | Tooling wiring only; no checkpoint, model score, or ship gate was produced ([JSON](langsmith-telemetry-smoke-20260724.json)) |