feat(evaluation): migrate evaluation harnesses from playground

[miguelgfierro]

Problem

The first migration of evaluation code into fireflyframework_agentic/evaluation/ brought over too much infrastructure alongside the metrics: a CLI (flyeval), a five-gate framework (G1–G5), a registry/corpus/matcher pipeline, a scorecard renderer, champion persistence, run-config snapshotting, and statistical helpers. The actual value — the LLM judge metric functions — was buried under 10 supporting files.

The goal of this PR (revised) is to gut that infrastructure and keep only the measurement code.

What We Keep

All metric functions from both evaluation systems:

Flyradar G4:

• [D] deterministic: source_coverage, excerpt_fill_rate
• [E] embedding-based: semantic_recovery
• [J] LLM judge: faithfulness, numeric_temporal_fidelity, citation_relevance, nc_semantic_precision, fabricated_entity, contradiction, open_gap, actionability, severity_calibration, answer_relevancy, surface_deduplication, comparative_vs_champion (champion passed as optional parameter — no persistence)

Flycanon:

• Custom: contains_answer, addresses_question (median of N LLM calls per item)
• RAGAS: answer_correctness, answer_relevancy, faithfulness, context_recall, context_precision

Retrieval (lab/retrieval_metrics.py): unchanged.

What We Delete

From fireflyframework_agentic/evaluation/:

File	Reason
cli.py	flyeval CLI — experiment orchestration, not measurement
gates.py	G1–G5 gate framework — pipeline infrastructure
corpus.py	Corpus loader — pipeline infrastructure
registry.py	Registry management — pipeline infrastructure
matcher.py	Anchored matching utilities — pipeline infrastructure
scorecard.py	Scorecard renderer — reporting, not measurement
run_config_snapshot.py	Run config capture — pipeline infrastructure
models.py	EvalConfig, GateVerdict — only used by deleted files
stats.py	aa_band, aggregate_grounding — only used by deleted files
champion.py	Champion persistence — comparative_vs_champion accepts champion data as a parameter instead

Tests for deleted modules also removed: test_champion.py, test_gates.py, test_matcher.py, test_stats.py.

Target Package Layout

fireflyframework_agentic/evaluation/
├── __init__.py       # exports: EvalContext, AdvisoryReport, all metric functions
├── judge_client.py   # JudgeClient — async LLM scoring client (httpx.AsyncClient)
└── judge.py          # ALL metric functions + EvalContext + AdvisoryReport

Three files. No CLI. No gates. No registry.

Unified Interface

Every metric — flyradar [D], [E], [J], flycanon custom, and RAGAS — shares the same async signature:

async def metric_name(item: dict, ctx: EvalContext) -> float | None

item is a plain dict with a normalized schema:

{
    "question": str,
    "answer": str,
    "reference": str,
    "contexts": list[str],
    # flyradar extras (optional):
    "sources": list[str],
    "excerpts": list[str],
    # for comparative_vs_champion (optional):
    "champion_answer": str | None,
}

EvalContext is a Pydantic model carrying all dependencies:

class EvalContext(BaseModel):
    model_config = ConfigDict(arbitrary_types_allowed=True)

    client: JudgeClient                     # async LLM call client (all metrics)
    embedder: OllamaEmbedder | None = None  # [E] metrics + RAGAS embeddings
    runs: int = 3                           # flycanon multi-run median

No ragas_llm / ragas_embeddings — RAGAS metrics wrap ctx.client and ctx.embedder in LangChain adapters internally. Callers see one client, one embedder.

Composable type alias:

Metric = Callable[[dict, EvalContext], Awaitable[float | None]]

Example:

ctx = EvalContext(client=JudgeClient(model="claude-sonnet-4-6", api_key=KEY))
metrics: list[Metric] = [faithfulness, contains_answer, answer_correctness]
scores = await asyncio.gather(*[m(item, ctx) for m in metrics])

judge_client.py

Contains only JudgeClient — a thin async HTTP client for Anthropic/OpenAI/Ollama scoring calls:

class JudgeClient:
    async def chat_json(self, system: str, user: str, max_tokens: int = 200) -> dict: ...

Uses httpx.AsyncClient. Handles 429/5xx retry with Retry-After parsing. No embedding logic — embeddings come from embeddings/providers/ollama.py (existing async OllamaEmbedder). cosine_similarity imported from embeddings/similarity.py.

Dependencies

evaluation optional extra changes:

• Remove: scipy (only used by deleted stats.py)
• Add: ragas, langchain-anthropic, langchain-ollama
• Keep: numpy

No changes to embeddings/ — existing async OllamaEmbedder used as-is.

Test plan

• pytest tests/unit/evaluation/test_judge.py tests/unit/lab/test_retrieval_metrics.py — all passing
• Each metric callable independently with a mocked EvalContext
• asyncio.gather(*[m(item, ctx) for m in metrics]) composes correctly across families