@tangle-network/agent-eval

Measure agent behavior, compare changes on the same cases, and improve prompts or skills without showing the final test cases to the optimizer.

The evaluation path runs in your TypeScript process. Model calls happen only through the clients and agents you configure.

New to the package? Read concepts first — it takes five minutes and defines every word used here.

Looking for a measured result (a lift, a null, a parity verdict)? The canonical registry is evidence/ — machine-readable records, a generated index, and a freshness gate.

Install

pnpm add @tangle-network/agent-eval

Quickstart

This example is offline and complete. Copy it, run it, then replace the agent and the judge with your product code.

import { defineAgentEval } from '@tangle-network/agent-eval/contract'

interface SupportCase {
  id: string
  kind: 'support'
}

const evalKit = defineAgentEval<SupportCase, string>({
  scenarios: [
    { id: 'refund', kind: 'support' },
    { id: 'shipping', kind: 'support' },
    { id: 'cancel', kind: 'support' },
  ],
  agent: async (prompt, scenario) =>
    String(prompt).includes('ticket') ? `Ticket ${scenario.id}: on it.` : 'On it.',
  judge: {
    name: 'ticket-id',
    dimensions: [{ key: 'present', description: 'The answer includes the ticket id' }],
    score: ({ artifact, scenario }) => {
      const present = artifact.includes(scenario.id) ? 1 : 0
      return { dimensions: { present }, composite: present, notes: '' }
    },
  },
  baselineSurface: 'Answer politely.',
  expectUsage: 'off',
})

console.log((await evalKit.evaluate()).aggregates.byJudge)
console.log(
  (await evalKit.evaluate({ surface: 'Answer politely and cite the ticket id.' })).aggregates
    .byJudge,
)

Each call runs every case, records what the agent produced, applies the same judge, and returns score distributions.

Three words carry this example. A case is one task the agent must do. A surface is the value being changed: a prompt, a skill, or a serialized configuration. A judge is a function that scores one produced result.

expectUsage: 'off' is set because this agent makes no paid calls. The default, 'assert', fails a run whose cells report no cost receipt. Keep the default whenever real model calls happen.

Runnable copy: examples/evaluate-a-change.

Auditable optimization history

Optimization methods may return a bounded SearchHistoryReceipt over Eval's canonical hash-chained SearchLedger. Existing callers keep working and see missing-history coverage. Autonomous and publication-grade runs set searchHistoryPolicy: 'require-complete' to refuse an incomplete planned denominator before the untouched final cases are opened.

The receipt is a small proof envelope, not another event log. Exact candidates, attempts, failures, decisions, and missing ids remain in the ledger. See complete optimization search history.

Which Front Door

Every row is a function you call. Each links to a runnable example.

When to call it	What you give it	What you get back
defineAgentEval() — you changed a surface and must know whether it helped	cases, an agent, a judge, a starting surface	evaluate() for scores, improve() for a search plus a release decision
selfImprove() — you want candidate generation, scoring, and a release decision in one call	cases, an agent, a judge, a starting surface	a report, a winner surface, and a gateDecision
analyzeRuns() — the runs already happened and no agent needs to run again	RunRecord[]	an InsightReport: distributions, paired lift, judge agreement, cost, failure clusters
fromFeedbackTable() (example) / fromOtelSpans() (example) — your data is in a table or an OTel collector, not in RunRecord shape	source rows or spans	RunRecord[] ready for analyzeRuns()
planCampaignRun() / runCampaign() — you need direct control of the case grid, or you must see it before paying for it	cases, a dispatch function, judges, a run directory	a per-cell schedule, then a campaign result with cached cells
loadEvalFixtureScenarios() — agents should add cases as folders on disk	evals/<name>/PROMPT.md plus checks	Scenario[] for runCampaign()
compareOptimizationMethods() — two search methods must be compared at equal budget	methods, a starting surface, train, selection, and final cases	per-method final lift, intervals, pairwise contrasts, and cost
gepaOptimizationMethod() / skillOptOptimizationMethod() — official GEPA or Microsoft SkillOpt should own the search	an objective, a recipe or trainer, an optimizer budget	an optimization method for the comparison above
externalTextOptimizationMethod() — another package owns text search and you keep the scoring	the package identity, limits, and a run callback	the same, with the final cases never exposed
SurfaceProposer — candidate generation belongs to your product	a propose() function	candidates the campaign executes, scores, and gates
runProfileMatrix() — the same cases must run across models or profiles	axes of models and profiles, cases	one row per cell, with an explicit unknown model rather than an invented one
ExperimentTracker — a candidate must beat its parent across N repetitions	reps with scores, run ids, and evidence references	a KEEP / ITERATE / NOISE / REGRESSION verdict with git provenance
sealExperiment() / openSealedExperiment() — the result must convince someone who does not trust you	arms, an admission funnel, an estimand, an interval, a decision table	a hashed rule tree, and executors that can run no other rule
runEquivalenceCheck() / VERIFICATION_STRATEGIES — the work has no held-out test suite	a claim, two blind arms, an injected checker	a certification that names who vouched and how it can fail
AnalystRegistry.runExact() — a batch of runs failed and you need cited findings	recorded evidence, a declared analyst list	findings with evidence references, an execution plan, and a receipt
analyzeTraces() — you have one question about a recorded run ("what first caused this failure?")	stored traces, the question, a DSPy RLM engine with a cost cap	an answer, findings with evidence references, and the investigation trajectory
runAnalystBenchmark() — an analyst's accuracy must be measured, not assumed	labeled issues and exact span locations	scored findings, trace reads, model calls, tokens, cost, and runtime
deltaRepair() — a finding must be graded by executing the repair it proposes	a trajectory, an analyst finding, a sandbox	the repair's measured effect against a no-fix control
replayVerify() — you must know whether a recorded failure still reproduces	a recorded shell trajectory and its pinned image	a re-execution verdict and the divergences found
analyzeSupervisorRun() — a recursive or supervised run directory must be read	a run directory	counts that stay missing when a measurement is missing, never zero
plantByPerturbation() / seedPlants() / catchRate() — you must know whether the grader catches a wrong answer, not only how the work scored	a grading set, and claims the grader verified	items authored wrong by one value, a sealed manifest, then a catch rate that refuses rather than guessing
buildRlDataset() — scored runs should become training data	run records and preferences	reward, preference, and supervised rows

Configure Model Calls

Benchmarks, user drivers, executors, built-in judges, completion checkers, and judge adapters all take the same ChatClient. You own model execution, and Agent Eval never goes looking for a credential: it reads no environment variable to find one, and every transport is bound at the call site.

Bind a transport one of two ways. Pass a chat function you wrote:

import { createChatClient } from '@tangle-network/agent-eval'

const chat = createChatClient({
  transport: 'custom',
  defaultModel: 'openai/gpt-4.1',
  maximumAttempts: 3,
  chat: async (request, opts) => myProviderClient(request, opts),
})

Or name an OpenAI-compatible endpoint and hand over a credential as values, and Agent Eval drives POST {baseUrl}/chat/completions for you:

const chat = createChatClient({
  transport: 'openai-compatible',
  baseUrl: 'https://router.example/v1',   // ends at /v1; the path is ours to append
  apiKey: process.env.MY_ROUTER_KEY,      // or `bearer`, or `authHeader`
  defaultModel: 'claude-sonnet-4-6',
})

Prefer the second over hand-rolling a fetch loop. It carries the retry, degrade, and — load-bearing — the servedModel echo that assertServedModel and assertCrossFamilyServed read; a transport that omits that field makes both checks report unreported, so the cross-vendor rules they enforce measure nothing. baseUrl and one credential form are required arguments with no default and no fallback: a half-configured client is refused at construction rather than reaching an endpoint you did not name.

On Agent Runtime, profileChatClient({ profile, executor, context }) from @tangle-network/agent-runtime/kernel is that transport: every call runs one exact AgentProfile and reports its measured usage, retries, and served model identity. Use sandbox-sdk for Sandbox and mock in tests. A custom adapter must return a ChatResponse and declare maximumAttempts before a capped cost account can dispatch it.

ChatResponse carries the whole execution record across that boundary: the served model id, measured input/output/reasoning/cached tokens, billed USD or an explicit unknown, the finish reason, and the per-token log probabilities the expectation judge scores on.

The official GEPA and SkillOpt optimizers run through a Python bridge. Install commands, version pins, and the reason for each pin: GEPA, SkillOpt, and DSPy.

Entry Points

Import	Use
@tangle-network/agent-eval/contract	Define an evaluation, run it, improve it, and analyze existing runs.
@tangle-network/agent-eval/campaign	Campaigns, optimization methods, comparisons, storage, and release rules.
@tangle-network/agent-eval/experiment	Experiments as sealed objects: registered rules, funnels, estimands, refusals.
@tangle-network/agent-eval/analyst	Built-in and custom trace analysts, labeled comparison, costs, and reports.
@tangle-network/agent-eval/trace-repair	Grade one analyst finding by executing the repair it proposes.
@tangle-network/agent-eval/trajectory-replay	Re-execute a recorded shell trajectory and check whether its failure reproduces.
@tangle-network/agent-eval/traces	Store, replay, and inspect structured traces.
@tangle-network/agent-eval/reporting	Statistical comparisons and report rendering.
@tangle-network/agent-eval/supervisor-run	Read recursive run directories without collapsing missing measurements to zero; agent-eval supervisor-run report <runDir> prints one.
@tangle-network/agent-eval/meta-eval	Measure the grader itself: judge calibration, sentinels, and seeded known-wrong plants.
@tangle-network/agent-eval/profile-cell	Create and validate portable agent-profile identities.
@tangle-network/agent-eval/ledger-core	Append-only hash-chained journal with idempotent append and chain verification.
@tangle-network/agent-eval/benchmarks	Benchmark adapters and retrieval metrics.
@tangle-network/agent-eval/rl	Export rewards, preferences, and training rows.
@tangle-network/agent-eval/wire	HTTP and RPC schemas for other languages.
@tangle-network/agent-eval/adapters/http	Run campaign cells on remote workers over HTTP.

Use the root import for common primitives. Use a subpath when you want an explicit capability boundary.

Documentation

Question	Read
What do these words mean?	docs/concepts.md
Why does this package exist, and where is it going?	docs/charter.md
Which run* function do I want?	docs/eval-surface-map.md
How do I choose a candidate-generation method?	docs/campaign-proposers.md
What is in an InsightReport?	docs/insight-report.md
How do I register an experiment as a sealed object?	docs/experiment.md
How is something certified without an answer key?	docs/verification-strategies.md
Where does every verifier land its result?	docs/verdicts.md
Does the grader catch a claim that is known to be wrong?	docs/plants.md
How do I turn a coding-agent session log into runs?	docs/code-agent-intake.md
How do I score a string from another language?	docs/wire-protocol.md

The example index lists every runnable example.

Development

pnpm install
pnpm typecheck
pnpm typecheck:examples
pnpm test
pnpm build

Python compatibility tests use the locked dependencies:

cd clients/python
uv sync --frozen --extra dev --group gepa-release
AGENT_EVAL_EXPECT_GEPA_RELEASE=1 \
  uv run --frozen --extra dev --group gepa-release \
  pytest tests/test_gepa_release_compatibility.py tests/test_gepa_bridge.py

uv sync --frozen --extra dev --group skillopt-source --group gepa-source
uv run --frozen pytest

uv sync --frozen --extra dev --extra dspy
uv run --frozen pytest tests/test_dspy_metric.py

License

MIT.

Supervisor-run resource receipts

The /supervisor-run reader preserves named-resource measurements in economics.resourceRecords. Each record identifies its node and source within the normalized journal or terminal result. Journal row indices refer to parsed rows after reader normalization, not original file line numbers. The Markdown report renders each resource name, unit, amount, and completeness flag. Comparison cells retain those same records without combining them.

A false known flag means the amount is a recorded subtotal, not complete usage. Missing maps, explicit empty maps, and invalid fields remain distinct from measured zero. Parent settlements and terminal results can include child usage, so these records are not additive totals. The reporter reads evidence; it does not enforce budgets or infer missing measurements.