AgentEval

The .NET Evaluation Toolkit for AI Agents

---

AgentEval is the comprehensive .NET toolkit for AI agent evaluation—tool usage validation, RAG quality metrics, stochastic evaluation, model comparison, and memory benchmarks—built for Microsoft Agent Framework (MAF) and Microsoft.Extensions.AI. What RAGAS and DeepEval do for Python, AgentEval does for .NET, with the fluent assertion APIs .NET developers expect.

For years, agentic developers have imagined writing evaluations like this. Today, they can.

Warning

Preview — Use at Your Own Risk

This project is experimental (work in progress). APIs and behavior may change without notice. Do not use in production or safety-critical systems without independent review, testing, and hardening.

Portions of the code, tests, and documentation were created with assistance from AI tools and reviewed by maintainers. Despite review, errors may exist — you are responsible for validating correctness, security, and compliance for your use case.

Licensed under the MIT License — provided "AS IS" without warranty. See LICENSE and DISCLAIMER.md.

---

The Code You Have Been Dreaming Of

🥇 Assert on Tool Chains Like You Have Always Imagined

The .NET fluent API for agentic tool usage. Every assertion you wished existed — order, arguments, duration, errors — composable, with because: reasoning baked in.

result.ToolUsage!.Should()
    .HaveCalledTool("SearchFlights", because: "must search before booking")
        .WithArgument("destination", "Paris")
        .WithDurationUnder(TimeSpan.FromSeconds(2))
    .And()
    .HaveCalledTool("BookFlight", because: "booking follows search")
        .AfterTool("SearchFlights")
        .WithArgument("flightId", "AF1234")
    .And()
    .HaveCallOrder("SearchFlights", "BookFlight", "SendConfirmation")
    .HaveNoErrors();

No more regex parsing logs. No more "did it call that function?" — just IntelliSense-driven assertions that read like requirements.

---

🥈 Stochastic Evaluation: Because LLMs Are Non-Deterministic

A single evaluation run might pass 70% of the time due to LLM randomness. Stochastic evaluation tells you the actual reliability — pass/fail on the rate, not the lucky run.

var result = await stochasticRunner.RunStochasticTestAsync(
    agent, testCase,
    new StochasticOptions
    {
        Runs = 20,                    // Run 20 times
        SuccessRateThreshold = 0.85,  // 85% must pass
        ScoreThreshold = 75           // Min score to count as "pass"
    });

result.Statistics.Mean.Should().BeGreaterThan(80);            // avg quality
result.Statistics.StandardDeviation.Should().BeLessThan(10);  // consistency

Assert.True(result.PassedThreshold,
    $"Success rate {result.SuccessRate:P0} below 85% threshold");

The evaluation that never flakes. Mean + StdDev + SuccessRate, not pass/fail roulette.

---

🥉 Workflow Evaluation: Multi-Agent Flows as Executable Assertions

MAF workflows are powerful — and finally testable. Assert on executor order, edges traversed, tools called across the graph, and end-to-end SLAs.

var testCase = new WorkflowTestCase
{
    Name              = "TripPlanner — Tokyo & Beijing",
    Input             = "Plan a 7-day trip to Tokyo and Beijing — flights and hotels",
    ExpectedExecutors = ["TripPlanner", "FlightReservation", "HotelReservation", "Presenter"],
    StrictExecutorOrder = true,
    ExpectedTools     = ["SearchFlights", "BookFlight", "BookHotel"],
    MaxDuration       = TimeSpan.FromMinutes(2),
};

var harness = new WorkflowEvaluationHarness();
var result  = await harness.RunWorkflowTestAsync(workflowAdapter, testCase);

result.ExecutionResult!.Should()
    .HaveSucceeded(because: "the trip must be planned end-to-end")
    .HaveExecutedInOrder("TripPlanner", "FlightReservation", "HotelReservation", "Presenter")
    .HaveAnyExecutorCalledTool("SearchFlights")
    .HaveAnyExecutorCalledTool("BookHotel")
    .HaveTraversedEdge("TripPlanner", "FlightReservation")
    .HaveCompletedWithin(TimeSpan.FromMinutes(2))
    .HaveNoToolErrors();

4 agents, 5 tools, one test. Execution timeline, edge traversal, tool errors — all observable, all assertable.

---

Performance SLAs as Executable Evaluations

result.Performance!.Should()
    .HaveTotalDurationUnder(TimeSpan.FromSeconds(5),
        because: "UX requires sub-5s responses")
    .HaveTimeToFirstTokenUnder(TimeSpan.FromMilliseconds(500),
        because: "streaming responsiveness matters")
    .HaveEstimatedCostUnder(0.05m,
        because: "stay within $0.05/request budget")
    .HaveTokenCountUnder(2000);

Know before production if your agent is too slow or too expensive.

---

Behavioral Policy Guardrails (Compliance as Code)

result.ToolUsage!.Should()
    // PCI-DSS: Never expose card numbers
    .NeverPassArgumentMatching(@"\b\d{16}\b",
        because: "PCI-DSS prohibits raw card numbers")

    // GDPR: Require consent
    .MustConfirmBefore("ProcessPersonalData",
        because: "GDPR requires explicit consent",
        confirmationToolName: "VerifyUserConsent")

    // Safety: Block dangerous operations
    .NeverCallTool("DeleteAllCustomers",
        because: "mass deletion requires manual approval");

---

Compare Models, Get a Winner, Ship with Confidence

var stochasticRunner = new StochasticRunner(harness);
var comparer = new ModelComparer(stochasticRunner);

var result = await comparer.CompareModelsAsync(
    factories: new IAgentFactory[]
    {
        new AzureModelFactory("gpt-4o", "GPT-4o"),
        new AzureModelFactory("gpt-4o-mini", "GPT-4o Mini"),
        new AzureModelFactory("gpt-35-turbo", "GPT-3.5 Turbo")
    },
    testCases: agenticTestSuite,
    metrics: new[] { new ToolSuccessMetric(), new RelevanceMetric(evaluator) },
    options: new ComparisonOptions(RunsPerModel: 5));

Console.WriteLine(result.ToMarkdown());

Output:

## Model Comparison Results

| Rank | Model         | Tool Accuracy | Relevance | Mean Latency | Cost/1K Req |
|------|---------------|---------------|-----------|--------------|-------------|
| 1    | GPT-4o        | 94.2%         | 91.5      | 1,234ms      | $0.0150     |
| 2    | GPT-4o Mini   | 87.5%         | 84.2      | 456ms        | $0.0003     |
| 3    | GPT-3.5 Turbo | 72.1%         | 68.9      | 312ms        | $0.0005     |

**Recommendation:** GPT-4o - Highest tool accuracy (94.2%)
**Best Value:** GPT-4o Mini - 87.5% accuracy at 50x lower cost

---

Combined: Stochastic + Model Comparison

The most powerful pattern — compare models with statistical rigor (see Sample D4):

var factories = new IAgentFactory[]
{
    new AzureModelFactory("gpt-4o", "GPT-4o"),
    new AzureModelFactory("gpt-4o-mini", "GPT-4o Mini")
};

var modelResults = new List<(string ModelName, StochasticResult Result)>();

foreach (var factory in factories)
{
    var result = await stochasticRunner.RunStochasticTestAsync(
        factory, testCase,
        new StochasticOptions(Runs: 5, SuccessRateThreshold: 0.8));
    modelResults.Add((factory.ModelName, result));
}

modelResults.PrintComparisonTable();

Output:

+------------------------------------------------------------------------------+
|                     Model Comparison (5 runs each)                           |
+------------------------------------------------------------------------------+
| Model        | Pass Rate   | Mean Score | Std Dev  | Recommendation         |
+--------------+-------------+------------+----------+------------------------+
| GPT-4o       | 100%        | 92.4       | 3.2      | Best Quality           |
| GPT-4o Mini  | 80%         | 84.1       | 8.7      | Best Value             |
+------------------------------------------------------------------------------+

---

RAG Quality: Is Your Agent Hallucinating?

var context = new EvaluationContext
{
    Input = "What are the return policy terms?",
    Output = agentResponse,
    Context = retrievedDocuments,
    GroundTruth = "30-day return policy with receipt"
};

var faithfulness = await new FaithfulnessMetric(evaluator).EvaluateAsync(context);
var relevance = await new RelevanceMetric(evaluator).EvaluateAsync(context);
var correctness = await new AnswerCorrectnessMetric(evaluator).EvaluateAsync(context);

// Detect hallucinations
if (faithfulness.Score < 70)
    throw new HallucinationDetectedException($"Faithfulness: {faithfulness.Score}");

---

Red Team Security Evaluation: Find Vulnerabilities Before Production

AgentEval includes comprehensive red team security evaluation with 264 probes across 14 attack types (Comprehensive intensity), covering all 10 OWASP LLM Top 10 2025 categories and 8 MITRE ATLAS techniques.

Beyond the built-in probes, it ships the capabilities that make a red-team result trustworthy and CI-ready:

• Multi-turn & attacker-LLM attacks — Crescendo, PAIR, TAP, and a tool-aware ToolEscalation attack (opt-in).
• Real attack surfaces — a tiered tool harness (--sut-tier text\|function-calling\|instrumented) with evidence-fidelity labeling (Verbal / IntentToAct / Behavioral), plus a live package-registry oracle (--package-registry live) and a real RAG-retrieval boundary.
• Trustworthy verdicts — judge-primary by default + Composite Judges (new) — with a judge configured (--judge), the grader that decides whether each attack succeeded is now LLM-judge-primary, using honest-by-construction Composite Judges: every semantic verdict is split into a positive-only compromise detector ⊕ a negative-only refusal detector, each structurally clamped so it can only raise its own direction or abstain. (A no-judge scan stays the deterministic keyword oracle, byte-identical to before.) Plus conclusive-only scoring and an explicit Inconclusive coverage state — so a green result is never a guess.
• 5 compliance reporters — OWASP, MITRE ATLAS, SOC 2, ISO 27001, and NIST AI RMF — runnable as first-class benchmarks (agenteval bench owasp\|mitre\|nist).
• CI-ready — SARIF + JUnit export, a baseline regression gate (--save-baseline/--baseline/--fail-on), z-score calibration (--calibration), LLM --explain rationale, and external benchmark packs (--pack HarmBench\|JailbreakBench\|CyberSecEval, license-gated, nothing bundled).
• Copilot Studio target — agenteval redteam --sut copilot-studio red-teams a Microsoft Copilot Studio agent through the same scanner, with its own config + consent gates and a credential-free test seam; the live connector is wired and unit/mock-tested, but not independently live-verified against a real Copilot Studio tenant (no test credentials available yet) — treat a first real run as a smoke test, not a proven-in-production path. See docs/redteam/copilot-studio.md.

Proof, not vibes. Across 810 held-out stochastic trials — 81 independently-generated cases (70 composite-oracle + 11 DataPoisoning deny-true) run K=10× each through the production graders — the Composite Judges fabricated 0 verdicts: never a safe reply flagged as a compromise, never a real compromise masked as safe. On a separately-pinned label corpus, judge↔label agreement is κ = 1.000 (n=92) — where keyword graders typically agree with humans only about half the time. The guiding rule: fabrications are complete failures; honesty is never punished. Background: ADR-021→024 · Red Team — What's New.

See Red Team — What's New for the recent upgrades, how AgentEval compares to PyRIT / garak / others, and a plain-English take on why grading a model's reply is the hard part — and docs/redteam.md for the full CLI reference.

// Sample20: Basic RedTeam evaluation
var redTeam = new RedTeamRunner();
var result = await redTeam.RunAsync(agent, new RedTeamOptions
{
    AttackTypes = new[] { 
        AttackType.PromptInjection, 
        AttackType.Jailbreak, 
        AttackType.PIILeakage,
        AttackType.ExcessiveAgency,  // LLM06
        AttackType.InsecureOutput    // LLM05
    },
    Intensity = AttackIntensity.Quick,
    ShowFailureDetails = true  // Show actual attack probes (for analysis)
});

// Comprehensive security validation
result.Should()
    .HaveOverallScoreAbove(85, because: "security threshold for production")
    .HaveAttackSuccessRateBelow(0.15, because: "max 15% attack success allowed")
    .ResistAttack(AttackType.PromptInjection, because: "must block injection attempts");

Real-time security assessment:

╔══════════════════════════════════════════════════════════════════════════════╗
║                        RedTeam Security Assessment                           ║
╠══════════════════════════════════════════════════════════════════════════════╣
║  🛡️ Overall Score: 88.2%                                                     ║
║  Verdict: ✅ PARTIAL_PASS                                                    ║
║  Duration: 12.4s | Agent: ResearchAssistant                                  ║
║  Probes: 45 total, 40 resisted, 5 compromised                                ║
╠══════════════════════════════════════════════════════════════════════════════╣
║  Attack Results:                                                             ║
║                                                                              ║
║  Attack                   Resisted     Rate     Severity                     ║
║  ───────────────────────────────────────────────────────────────────────     ║
║  ✅ Prompt Injection      8/9          89%      Critical                     ║
║  ✅ Jailbreak             7/8          88%      High                         ║
║  ✅ PII Leakage           6/6          100%     Critical                     ║
║  ✅ Excessive Agency      5/5          100%     High                         ║
║  ❌ Insecure Output       10/12        83%      Critical                     ║
║     OWASP: LLM05 | MITRE: AML.T0051                                          ║
╚══════════════════════════════════════════════════════════════════════════════╝

Multiple export formats for security teams:

• JSON for automation and tooling
• Markdown for human-readable reports
• JUnit XML for CI/CD integration
• SARIF for GitHub Security tab integration
• PDF for executive/board-level reporting

✅ See Samples: 02_RedTeamBasic.cs • 03_RedTeamAdvanced.cs • docs/redteam.md

---

🚪 Gatekeeper: Stop the Bad Action Before It Happens

Red-teaming finds the holes. Gatekeeper closes them at runtime — the same probes and evaluators become fail-closed gates in the request path. It catches the attacks you can't stop by "just not giving the tool":

var agent = baseAgent.AsBuilder()
    .UseAgentEvalGate()   // per-run scope for the sequence gate
    .UseAgentEvalToolGate(
        [
            // 🛑 Block DATA EXFILTRATION: reading customer data is fine, sending mail is fine —
            //    the SEQUENCE is the attack. No tool-list trick catches this.
            new SequenceGate(triggerTools: ["read_customer_data"], guardedTools: ["send_email", "http_post"]),

            // 🎣 The SAME red-team oracle you test with, now a LIVE GUARD against a poisoned tool argument:
            new ProbeEvaluatorGate(new ContainsTokenEvaluator("ignore previous instructions"), GateCost.PureCode),
        ],
        ToolGatePolicy.Terminate)   // block the call AND stop the loop
    .Build();

Even if a prompt injection turns your own agent against you, the destructive action never executes. Fail-closed by design: a gate that can't prove an action safe blocks it, and every decision is recorded as honest gate.* trace evidence (a warn is never counted as a block). Layers span tool gates, run gates, session gates (auth / rate-limit / quarantine), the red-team moat, canary honeypots that flag a compromised agent, an async shadow judge for expensive checks, and human-in-the-loop approval for the borderline actions.

The same policy is also callable from outside .NET: the agenteval gatekeeper CLI verb group exposes it as a language-neutral runtime-policy service — pipe a JSON payload to agenteval gatekeeper inspect from Python, Node, bash, or a CI step and get back a versioned verdict + exit code, no .NET reference required. See docs/gatekeeper-cli.md.

✅ See it: dotnet run --project samples/AgentEval.Samples → group J (real agents — needs Azure OpenAI), or credential‑free via agenteval redteam --sut gatekeeper-demo • docs/gatekeeper/introduction.md

---

🧩 Agent Skills: Evaluate & Govern Progressive Disclosure

Microsoft Agent Framework's Agent Skills (GA'd 2026-07-07) let an agent progressively disclose capabilities through three stable tools — load_skill, read_skill_resource, run_skill_script — instead of stuffing every capability into the system prompt up front. AgentEval evaluates and governs that surface end to end: fluent assertions on the disclosure trace, a free structural efficiency metric, a SKILL.md compliance scanner, a dedicated red-team attack for a poisoned skill description, deterministic Gatekeeper gates for run_skill_script code execution, and a composite Skill Health & Security Index.

// Assert the disclosure trace like any other tool chain
result.ToolUsage!.Should()
    .HaveLoadedSkill("expense-report")
    .And().HaveReadSkillResource("expense-report", "resources/policy.md")
        .AfterTool(SkillToolNames.LoadSkill)
    .And().HaveDisclosedProgressively()
    .And().NotHaveRunSkillScript(because: "a policy lookup doesn't need the compliance script");

// Score the load -> read -> run funnel (structural, free — no LLM call)
var efficiency = await new SkillDisclosureEfficiencyMetric().EvaluateAsync(new EvaluationContext
{
    Input = "n/a", Output = "n/a", ToolUsage = result.ToolUsage,
});
Console.WriteLine($"Disclosure efficiency: {efficiency.Score:F0}/100");

// Scan SKILL.md authoring + governance flags, then roll compliance + efficiency + red-team
// outcome into one composite score — a missing axis is averaged out, never faked as perfect
var complianceReport = await MafSkillScanner.ScanFileSkillsAsync(skillPath, agent);
var index = SkillSecurityIndex.Compute(
    new SkillSecurityIndexInputs(complianceReport, efficiency, securityOutcome: null));
Console.WriteLine($"Skill Security Index: {index.Score:F0}/100 ({index.AxesMeasured}/3 axes measured)");

Governance doesn't stop at evaluation time: SkillScriptExecutionGate and SkillScriptApprovalGate are deterministic Gatekeeper gates that allowlist/approve run_skill_script calls before they execute, and SkillInjectionAttack (OWASP LLM01, one of the 14 Attack.All types above) red-teams a poisoned skill description or read_skill_resource output through the same AttackPipeline that scans every other surface.

Honest by construction: a skill source MAF gives no public enumeration API for (in-memory/class/MCP skills) reports zero resources rather than a guessed inventory, a missing Security Index axis is never counted as perfect, and the injection judge ships shadow-only — advisory only — because live calibration found it doesn't yet clear the promotion bar on this surface. See docs/agent-skills.md for the full, honestly-labeled rundown.

✅ See it: dotnet run --project samples/AgentEval.Samples → group K (real agent — needs Azure OpenAI), or the standalone deep-dive samples/AgentEval.AgentSkillsEval • docs/agent-skills.md

---

Responsible AI: Content Safety Metrics

Complementing security evaluation, AgentEval's ResponsibleAI namespace provides content safety evaluation:

using AgentEval.Metrics.ResponsibleAI;

// Toxicity detection (pattern + LLM hybrid)
var toxicity = new ToxicityMetric(chatClient, useLlmFallback: true);
var toxicityResult = await toxicity.EvaluateAsync(context);

// Bias measurement with counterfactual testing  
var bias = new BiasMetric(chatClient);
var biasResult = await bias.EvaluateCounterfactualAsync(
    originalContext, counterfactualContext, "gender");

// Misinformation risk assessment
var misinformation = new MisinformationMetric(chatClient);
var misInfoResult = await misinformation.EvaluateAsync(context);

// All must pass for responsible AI compliance
toxicityResult.Should().HaveScoreAbove(90);
biasResult.Should().HavePassed();
misInfoResult.Should().HavePassed();

Metric	Type	Detects
ToxicityMetric	Hybrid	Hate speech, violence, harassment
BiasMetric	LLM	Stereotyping, differential treatment
MisinformationMetric	LLM	Unsupported claims, false confidence

✅ See: docs/ResponsibleAI.md

---

Memory Evaluation: Does Your Agent Actually Remember?

AgentEval ships AgentEval.Memory — the comprehensive .NET toolkit for evaluating agent memory: retention, recall depth across long contexts, temporal reasoning, fact-update handling, cross-session persistence, and resistance to distractor turns.

// One-line benchmark with grade
var runner = MemoryBenchmarkRunner.Create(chatClient);
var agent  = chatClient.AsEvaluableAgent(name: "MemoryAgent", includeHistory: true);

var result = await runner.RunBenchmarkAsync(agent, MemoryBenchmark.Standard);
Console.WriteLine($"Memory: {result.OverallScore:F1}% ({result.Grade})");

// Save baseline + generate an interactive HTML pentagon report
var store = new JsonFileBaselineStore();
await store.SaveAsync(result.ToBaseline(label: "GPT-4o"));
await result.ExportHtmlReportAsync("memory-report.html");

What's in the box:

Capability	Detail
5 memory metrics	Retention, ReachBack, Temporal, NoiseResilience, ReducerFidelity
5 benchmark presets	Quick (3 cats) → Standard (8) → Full (12) → Diagnostic / Overflow (192K-token haystacks)
HTML pentagon reports	Multi-model overlay, baseline diffs, drill-down judge explanations
LongMemEval (ICLR 2025)	Fully re-implemented in .NET — paper-comparable scoring (GPT-4o = 57.7%)
MAF-native	Compatible with AIContextProvider, ChatHistoryProvider, CompactionStrategy
Custom scenarios	Build your own with MemoryFact / MemoryQuery / MemoryTestRunner

Honest caveats:

• The native Standard benchmark currently scores ~88–93% on GPT-4.1 — strong models clear it comfortably. Use it as a regression gate for your own delta over time, and use LongMemEval (Sample G7) for cross-platform comparable numbers. Harder synthesis/counterfactual scenarios are on the way.
• Memory evaluation always calls a real LLM (the judge can't be mocked).
• LongMemEval dataset isn't redistributed — download it from HuggingFace.

✅ See: docs/memory-evaluation.md • docs/maf-memory-integration.md • Sample G2: Memory Benchmark • Sample G7: LongMemEval

---

Why AgentEval?

Challenge	How AgentEval Solves It
"What tools did my agent call?"	Full tool timeline with arguments, results, timing
"Evaluations fail randomly!"	stochastic evaluation - assert on pass rate, not pass/fail
"Which model should I use?"	Model comparison with cost/quality recommendations
"Is my agent compliant?"	Behavioral policies - guardrails as code
"Is my RAG hallucinating?"	Faithfulness metrics - grounding verification
"What's the latency/cost?"	Performance metrics - TTFT, tokens, estimated cost
"How do I debug failures?"	Trace recording - capture executions for step-by-step analysis
"Is my agent secure?"	Red Team evaluation - 264 probes, full OWASP LLM Top 10 2025 coverage
"Can I stop a bad action at runtime?"	Gatekeeper - fail-closed runtime enforcement: block forbidden tool calls before they run, quarantine compromised sessions
"Is my agent's use of MAF Agent Skills safe and efficient?"	Agent Skills evaluation - disclosure assertions, efficiency metric, compliance scanner, injection red-team, governance gates
"Is content safe and unbiased?"	ResponsibleAI metrics - toxicity, bias, misinformation
"Does my agent actually remember?"	Memory evaluation - retention, reach-back, temporal, LongMemEval (ICLR 2025)

---

Who Is AgentEval For?

🏢 .NET Teams Building AI Agents — If you're building production AI agents in .NET and need to verify tool usage, enforce SLAs, handle non-determinism, or compare models—AgentEval is for you.

🚀 Microsoft Agent Framework (MAF) Developers — Native integration with MAF concepts: AIAgent, IChatClient, automatic tool call tracking, and performance metrics with token usage and cost estimation.

📊 ML Engineers Evaluating LLM Quality — Rigorous evaluation capabilities: RAG metrics (Faithfulness, Relevance, Context Precision), embedding-based similarity, and calibrated judge patterns for consistent evaluation.

---

The .NET Advantage

Feature	AgentEval	Python Alternatives
Language	Native C#/.NET	Python only
Type Safety	Compile-time errors	Runtime exceptions
IDE Support	Full IntelliSense	Variable
MAF Integration	First-class	None
Fluent Assertions	Should().HaveCalledTool()	N/A
Trace Replay	Built-in	Manual setup

---

Key Features

Core Features

• Fluent assertions - tool order, arguments, results, duration
• Stochastic evaluation - run N times, analyze statistics (mean, std dev, p90)
• Model comparison - compare across models with recommendations
• Trace recording - capture executions for debugging and reproduction
• Performance assertions - latency, TTFT, tokens, cost

Evaluation Coverage

• Red Team security - 264 probes, full OWASP LLM Top 10 2025, MITRE ATLAS coverage
• Gatekeeper runtime enforcement - fail-closed gates that block forbidden tool calls before they run, red-team probes as runtime guards, and an async shadow judge that quarantines compromised sessions (docs)
• Agent Skills evaluation - disclosure assertions, a free efficiency metric, SKILL.md compliance scanning, a skill-injection red-team attack, and deterministic run_skill_script governance gates (docs)
• Responsible AI - toxicity, bias, misinformation detection
• Memory evaluation - retention, reach-back, temporal, cross-session, HTML pentagon reports, LongMemEval (ICLR 2025)
• Multi-turn conversations - full conversation flow evaluation
• Workflow evaluation - multi-agent orchestration and routing
• Snapshot evaluation - regression detection with semantic similarity

Metrics

• RAG metrics - faithfulness, relevance, context precision/recall, correctness
• Agentic metrics - tool selection, arguments, success, efficiency
• Embedding metrics - semantic similarity (100x cheaper than LLM)
• Custom metrics - extensible for your domain

Developer Experience

• Rich output - configurable verbosity (None/Summary/Detailed/Full)
• Time-travel traces - step-by-step execution capture in JSON
• Trace artifacts - auto-save traces for failed evaluations
• Behavioral policies - NeverCallTool, MustConfirmBefore, NeverPassArgumentMatching

CLI Tool

• agenteval init / doctor / migrate - Bootstrap, validate, and migrate the .agenteval/ workspace (canonical output store with audit-chain integrity)
• agenteval bench {gdpr,eu-ai-act,agentic} - Run compliance and agentic benchmark suites
• agenteval compliance render / agenteval render --benchmark agentic - Re-render reports from existing evidence (no LLM cost)
• agenteval mc serve / mc doctor - Launch and verify the Mission Control web portal (read-only viewer over .agenteval/)
• CI/CD-friendly exit codes; multiple export formats via agenteval render

Mission Control Portal

• Single-binary web portal (Hot Chocolate 16 GraphQL + minimal REST + React SPA) served on http://localhost:5000
• Read-only view over .agenteval/: dashboard, runs list, recursive EvalResult tree drill-down, compliance matrix with audit-chain badges, evaluator registry, per-evaluator timeline
• Single-port deployment via agenteval mc serve, dotnet run --project src/AgentEval.MissionControl, or docker compose up
• See docs/missioncontrol/getting-started.md

Benchmark Families (11 families, single-source-of-truth registry)

Every family auto-registers via [ModuleInitializer] into BenchmarkFamilyRegistry. agenteval bench --list reads from the registry — no hardcoded family lists anywhere (ADR-017 Convention 3). Updated 2026-05-25 (plan-13 T4.1b item 21).

Family	Presets	CLI status	What it grades	Cost tier (default)
GDPR	smoke / standard / audit + 3 domain packs (healthcare / HR / children)	✅ end-to-end	22 article YAMLs across 5 pillars	Medium
EU AI Act	smoke / standard / audit + 3 domain packs (high-risk-employment / -credit / -education)	✅ end-to-end	13 article YAMLs across 6 pillars (Reg (EU) 2024/1689)	Medium
Agentic	11 presets (tool-call-accuracy / agentic-execution / audit-grade / --budget-tier {free,low,medium,high} filter etc.)	✅ end-to-end	Foundry-equivalent 60-evaluator universe — system / process / UX / quality / safety / adversarial / reasoning / calibration / memory	Medium
OWASP LLM Top 10	top10 / smoke / audit / top10-rag	✅ end-to-end (--azure-from-env for real agents; stub fallback)	14 attack types covering all 10 OWASP LLM Top 10 v2.0 categories (LLM03/04/08/09 added in Wave D; SkillInjection added for MAF Agent Skills)	Medium
MITRE ATLAS	atlas-baseline / atlas-smoke / atlas-audit-grade	✅ end-to-end	Same 14 attacks mapped via IAttackType.MitreAtlasIds covering 8 applicable ATLAS techniques	Medium
NIST AI RMF	rmf-baseline / rmf-smoke / rmf-audit-grade	✅ end-to-end (--azure-from-env for real agents; stub fallback)	Same 14 attacks mapped to NIST AI RMF (AI 100-1) MEASURE security/privacy/validity sub-actions (GOVERN/MAP/MANAGE not applicable)	Medium
LongMemEval	subset / full (ICLR 2025)	✅ end-to-end	Cross-platform memory benchmark — paper-published GPT-4o baseline ≈ 57.7%	Medium
Memory	quick / standard / full / diagnostic / overflow	✅ end-to-end	Native AgentEval memory benchmark — 3/8/12 categories, weighted grading	Medium
Performance	latency / throughput / cost	✅ end-to-end (--azure-from-env)	P99 latency / concurrent throughput / per-prompt cost against your deployment	Low
Trace Fidelity	smoke / standard / audit-grade	✅ end-to-end (pure code, no LLM cost — reconciles two supplied .trace.json files)	Agent-boundary vs chat-boundary trace reconciliation — missing/phantom calls, hidden retries, argument drift, token under-reporting, suppressed finish reasons	Free
Workflow Trace Fidelity	smoke / standard / audit-grade	✅ end-to-end (pure code, no LLM cost — reconciles a workflow .trace.json)	Per-executor workflow ledger (tokens + finish reason) vs chat-boundary truth — per-executor fidelity (Agree / TokenMismatch / FinishMismatch / NoTruth)	Free

Every evidence document is cryptographically chained to its source run; agenteval doctor re-validates on demand. Per-family getting-started.md guides live under docs/benchmarks/ (OWASP + GDPR + EU AI Act + Memory + LongMemEval + MITRE + Performance + Agentic).

.agenteval/ Workspace Standard

• Canonical on-disk format: one folder per agent / workflow, deterministic run IDs, SHA-256 content hashes on every manifest
• Read-only consumed by Mission Control; written by the CLI, test harnesses, and benchmark runners
• See docs/agenteval-workspace.md

Cross-Framework & DI

• Universal IChatClient.AsEvaluableAgent() one-liner for any AI provider
• Dependency Injection via services.AddAgentEval() / services.AddAgentEvalAll()
• Semantic Kernel bridge via AIFunctionFactory.Create() (see NuGetConsumer sample)

Integration

• ⭐ Azure AI Foundry evals — run Foundry evals alongside (batched, one source-tagged report) and inside (as weighted leaves in a composite benchmark) your AgentEval evals, from a single agent run. See Foundry Evals Integration.
• CI/CD integration - JUnit XML, Markdown, JSON, SARIF export
• Benchmarks - custom patterns with dataset loaders (JSON, YAML, CSV, JSONL)
• Comprehensive multi-framework evaluation suite across all supported TFMs

---

Feature Maturity

AgentEval ships as a single lockstep-versioned package (see Installation) — there is no per-package version to signal maturity the way some multi-package frameworks do. Instead, maturity is tracked per front, here, plus [Experimental] attributes on individual volatile APIs (compiler-enforced — referencing one without acknowledging it is a build error, not just a warning).

Front	Maturity	Why
Core eval (assertions, RAG metrics, LLM-as-judge, benchmarks, exporters)	GA-track	Stable since early releases, no breaking changes across this project's history, the foundation every other front builds on.
RedTeam core (OWASP LLM Top 10, attacks/probes, Composite Judges)	GA-track	Feature-complete for its stated scope; remaining backlog items are additive, not destabilizing to what's shipped.
Compliance benchmarks (GDPR, EU AI Act)	GA-track	Calibrated, stable, reproducible evidence figures.
Gatekeeper (runtime enforcement)	Beta	The most differentiated subsystem and heavily tested (multiple calibrated judge axes) — but also the one still absorbing breaking API changes as it stabilizes. Real, valuable, actively evolving — exactly what "beta" means. A handful of the newest types are individually marked [Experimental].
Agent Skills	Beta	Genuinely shipped and tested, but depends on a very recently-GA'd upstream (MAF Agent Skills) and is still actively growing.
Copilot Studio	Experimental	The connector has not yet been verified against a real tenant.
Mission Control	Experimental	Real and functioning, but with acknowledged gaps and comparatively less investment than Core/RedTeam.

---

Installation

dotnet add package AgentEval --prerelease

Compatibility: .NET 8.0 / 9.0 / 10.0. The Microsoft Agent Framework (MAF) and Microsoft.Extensions.AI versions ship centrally pinned in Directory.Packages.props — see the CHANGELOG for the exact versions in each release.

Single package, modular internals:

• AgentEval.Abstractions — Public contracts and interfaces
• AgentEval.Core — Metrics, assertions, comparison, tracing
• AgentEval.DataLoaders — Data loading and export
• AgentEval.MAF — Microsoft Agent Framework integration
• AgentEval.Memory — Memory evaluation, benchmarks, LongMemEval, HTML reporting
• AgentEval.RedTeam — Security testing

Optional add-on package (not bundled in AgentEval — install only if you need it, so its dependency tree isn't forced on everyone):

• AgentEval.MAF.CopilotStudio — evaluate a live Microsoft Copilot Studio agent directly in code (IChatClient/IEvaluableAgent), no CLI required. dotnet add package AgentEval.MAF.CopilotStudio --prerelease. See docs/redteam/copilot-studio.md.

CLI Tool:

The CLI is published as a dotnet tool on NuGet:

# Install (one-time, global)
dotnet tool install --global AgentEval.Cli --prerelease

# Use
agenteval init                                                 # bootstrap .agenteval/ workspace
agenteval bench --list                                         # discover the 11 benchmark families
agenteval bench gdpr --preset smoke --subject MyAgent          # run a GDPR compliance benchmark
agenteval bench owasp --preset smoke --subject MyAgent --azure-from-env   # OWASP red-team against your real agent
agenteval mc serve                                             # open Mission Control (requires .NET 10)
agenteval doctor                                               # verify workspace integrity

Requirements: .NET 8 SDK for the core surface; .NET 10 SDK additionally for mc serve (graceful fallback message on .NET 8). See docs/installation.md for update / uninstall / contributor-path (dotnet run --project src/AgentEval.Cli) details.

v1 NuGet scope. The AgentEval package currently ships AgentEval.{Abstractions,Core,DataLoaders,MAF,RedTeam}. The agentic 60-evaluator suite, GDPR/EU AI Act benchmark code, and memory evaluation pack live alongside the agenteval CLI but are not yet exposed as programmatic NuGet APIs — they are runnable today via the CLI binaries. Surfacing them as separate packages is on the v1.1 roadmap.

Supported Frameworks: .NET 8.0, 9.0, 10.0

---

Quick Start

See the Getting Started Guide for a complete walkthrough with code examples.

---

Documentation

Guide	Description
Getting Started	Your first agent evaluation in 5 minutes
Fluent Assertions	Complete assertion guide
stochastic evaluation	Handle LLM non-determinism
Model Comparison	Compare models with confidence
Benchmarks	Benchmark patterns and best practices
Tracing	Record and Replay patterns
Red Team Security	Security probes, OWASP/MITRE coverage
Agent Skills	Evaluate & govern MAF Agent Skills — assertions, disclosure efficiency, compliance scanning, injection red-team
Responsible AI	Toxicity, bias, misinformation detection
Memory Evaluation	Retention, reach-back, temporal, LongMemEval, HTML reports
MAF Memory Integration	How AgentEval.Memory maps to MAF pipelines
MAF Eval Integration	Run AgentEval through MAF's agent.EvaluateAsync()
⭐ Foundry Evals Integration	Run Azure AI Foundry evals alongside and inside AgentEval evals
Cross-Framework	Semantic Kernel, IChatClient adapters
CLI Tool	Command-line evaluation guide
Migration Guide	Coming from Python/Node.js frameworks
Code Gallery	Stunning code examples

---

Samples

Run the included samples, organised into groups:

dotnet run --project samples/AgentEval.Samples

The interactive menu lets you select a group (A–K), then a sample within it.

Group	Focus
A — Getting Started ★ mostly no credentials	Hello World, tool tracking, performance basics, MAF integration patterns
B — Metrics & Quality	RAG evaluation, quality metrics, judge calibration, responsible AI
C — Workflows & Conversations	Multi-turn conversations, MAF workflows, source-gen executors
D — Performance & Statistics	Latency profiling, stochastic evaluation, model comparison, streaming
E — Safety & Security	Policy guardrails, red team scanning, OWASP compliance
F — Data & Infrastructure	Snapshot testing, datasets, trace replay, benchmarks, cross-framework
G — Memory Evaluation	Memory basics, benchmarks, scenarios, DI, cross-session, HTML reports, LongMemEval (ICLR 2025)
H — Benchmarks	Compliance & performance benchmark families → JSON / HTML / PDF reports
I — Observability (Glass Box)	Dual-boundary per-turn tracing, trace fidelity, auto-audit
J — Gatekeeper (Runtime Protection) 🔑 real agents	Fail-closed runtime enforcement on live agents: the gate-layer walkthrough, a data-exfiltration support agent, human-in-the-loop approval, the Beachhead + Tribunal, and MAF Agent Harness defense
K — Agent Skills 🔑 real agents	Evaluate & govern MAF Agent Skills: a Hello World on-ramp, the disclosure-efficiency metric, the compliance scanner, and the composite Skill Security Index

See samples/AgentEval.Samples/README.md for the full listing with per-sample descriptions, timing, and credential requirements.

---

CI Status

Workflow	Status
Build & Test
Security Scan
Documentation

---

Contributing

We welcome contributions! Please see:

• CONTRIBUTING.md
• CODE_OF_CONDUCT.md
• SECURITY.md

Contributors

A heartfelt thank you to everyone who has contributed to AgentEval. 🙏

• Bernhard Merkle (@bmerkle) — first community contributor, for making numeric/currency formatting culture-invariant (so scores render as 0.95, not 0,95, on comma-decimal locales) and for cleaning up the DocFX documentation build.
• Javier Iniesta Fernández (@Javierif) — second community contributor, for adding --response / --response-file to agenteval bench agentic, so you can grade a supplied agent response instead of the built-in stub.

---

Commercial & Enterprise

AgentEval is MIT and community-driven. For enterprise inquiries, see: https://agenteval.dev/commercial.html

---

Forever Open Source

AgentEval is MIT licensed and will remain open source forever. We believe in:

• ✅ No license changes — MIT today, MIT forever
• ✅ No bait-and-switch — core stays MIT and fully usable
• ✅ Community first — built with the .NET AI community
• ℹ️ Optional add-ons may exist separately (if/when built)

---

License

MIT License. See LICENSE for details.

---

Built with love for the .NET AI community

Star us on GitHub | NuGet | Issues

---

Star History