Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -319,6 +319,18 @@ instead of memorizing paths. Selected immutable snapshots publish to Git with
Every new run writes `outputs/runs/<id>/trace.json` and OTLP JSONL signals under
`outputs/traces/<trace-id>/`. Set `OTEL_EXPORTER_OTLP_ENDPOINT` for an optional
remote OTLP mirror; detailed domain traces remain local and linked by trace ID.
Set `LANGSMITH_TRACING=true` and provide `LANGSMITH_API_KEY` to export an
additional, best-effort aggregate trace to the `slm-training` LangSmith project
(`LANGSMITH_PROJECT`, `LANGSMITH_ENDPOINT`, and `LANGSMITH_WORKSPACE_ID` are
optional overrides). This exports run/suite metrics, version stamps, gate
verdicts, and AgentV summaries only—never prompts, targets, completions,
checkpoints, raw logs, or credentials. Local evidence and AgentEvals remain
authoritative when LangSmith is unavailable. Install the harness-side client
with `pip install -e '.[observability]'`; the shared trace boundary loads the
repository's ignored `.env` without overriding environment variables provided
by CI or a shell. Standard OTLP settings are honored as well:
`OTEL_SERVICE_NAME`, `OTEL_RESOURCE_ATTRIBUTES`, signal-specific endpoints,
and `OTEL_EXPORTER_OTLP_TIMEOUT` (milliseconds; bounded to five seconds).

The flush pipeline remains: curated seeds + RICO + Awwwards → deterministic
quality synth → per-record DESIGN.md + OpenUI validate → quality gates → stable
Expand Down
5 changes: 5 additions & 0 deletions docs/design/agentv-evaluation.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,10 @@ AgentEvals assertion results has `authority: "AgentEvals assertions"`.
under `<run-dir>/evals/`.
5. `write_ship_gates` projects those assertion results into the compatibility
`gates.json` shape and records the AgentEvals authority.
6. When opt-in LangSmith tracing is configured, the runner publishes its
aggregate AgentV summary as a child of the existing W3C-correlated run
trace. This is observational only; AgentEvals assertions, local artifacts,
and ship gates retain authority if export fails.

The `agentv` npm package is retained for the canonical CLI and dashboard. In
the pinned release its published package is CLI-only, so programmatic execution
Expand Down Expand Up @@ -82,3 +86,4 @@ The implementation check is recorded in
| --- | --- | --- | --- |
| 2026-07-14 | CPU, steps 0, no model backend; AgentV SDK fixture plus model/loss/task/train-loop harness tests | 34/34 focused tests passed; SDK fixture wrote valid AgentEvals JSONL and AgentV artifacts; dependency audit has 0 high/critical findings | Tooling wiring only; no checkpoint, model score, or ship gate was produced |
| 2026-07-23 | CPU, steps 0, no model backend; focused assertion-authority tests with the pinned SDK | AgentEvals JSONL carried required code-graders over raw criteria; the runner produced 1/1 passing fixture criteria with 0 execution errors; 135 focused gate/consumer checks and all 6 interpreted page validations passed. Dashboard production build remained environment-blocked because the locked `@openfeature/web-sdk` dependency was unavailable and automatic review rejected installation before execution. | Tooling and gate-authority wiring only; no checkpoint or model-quality claim |
| 2026-07-24 | Local summary-only telemetry configuration smoke; LangSmith `slm-training` project, two-second bounded flush, standard OTLP resource defaults | W3C root trace `fdf78e5360a6647b09fa0882a2c5d4ec` was published and read back as `slm.telemetry.config` | Tooling wiring only; no checkpoint, model score, or ship gate was produced ([JSON](langsmith-telemetry-smoke-20260724.json)) |
Loading