diff --git a/.github/workflows/pr-gate.yml b/.github/workflows/pr-gate.yml index dc60e00a..28519cfe 100644 --- a/.github/workflows/pr-gate.yml +++ b/.github/workflows/pr-gate.yml @@ -57,7 +57,7 @@ jobs: - uses: actions/setup-python@v6 with: python-version: '3.13' - - run: uv sync --extra dev --extra binary --extra vectorstores-sqlite-vec --extra openai-embeddings + - run: uv sync --extra dev --extra binary --extra vectorstores-sqlite-vec --extra openai-embeddings --extra evaluation - run: uv run pyright test: @@ -72,7 +72,7 @@ jobs: - uses: actions/setup-python@v6 with: python-version: '3.13' - - run: uv sync --extra dev --extra binary --extra vectorstores-sqlite-vec --extra vectorstores-pgvector --extra openai-embeddings --extra script-execution + - run: uv sync --extra dev --extra binary --extra vectorstores-sqlite-vec --extra vectorstores-pgvector --extra openai-embeddings --extra evaluation --extra script-execution - run: uv run pytest -m "not nightly" --cov --cov-report=term-missing build: diff --git a/README.md b/README.md index 341e81a0..c3ad27fc 100644 --- a/README.md +++ b/README.md @@ -318,6 +318,12 @@ create your own components; the framework discovers them via duck typing. `EvalDataset` loads/saves test cases from JSON. `ModelComparison` runs the same prompts across multiple agents for side-by-side analysis. +- **Evaluation** — LLM-as-judge metrics (faithfulness, relevancy, answer correctness, + RAGAS, …) and deterministic retrieval metrics (recall@k, MRR, MAP, nDCG, …) for + assessing LLM and pipeline outputs. Each metric is a plain function you call directly. + Install with `pip install "fireflyframework-agentic[evaluation]"`. + See [docs/evaluation.md](docs/evaluation.md) for the full guide. + > **Optional developer tooling.** `fireflyframework_agentic.experiments` (A/B > experiments) and `fireflyframework_agentic.lab` (offline evaluation / > benchmarking) are leaf modules — nothing in the core imports them and they add @@ -751,6 +757,7 @@ Detailed guides for each module: - [Secure Script Execution](docs/execution.md) — Deny-by-default Monty sandbox, static safety pre-screen, `SecureScriptRunner`, Firefly Code Mode - [Experiments](docs/experiments.md) — A/B testing, variant comparison - [Lab](docs/lab.md) — Benchmarks, datasets, evaluators +- [Evaluation](docs/evaluation.md) — LLM-as-judge metrics, RAGAS, retrieval metrics - Studio — moved to [fireflyframework-agentic-studio](https://github.com/fireflyframework/fireflyframework-agentic-studio) --- diff --git a/docs/evaluation.md b/docs/evaluation.md new file mode 100644 index 00000000..07a1a83f --- /dev/null +++ b/docs/evaluation.md @@ -0,0 +1,329 @@ +# Evaluation Guide + +Copyright 2026 Firefly Software Foundation. Licensed under the Apache License 2.0. + +The Evaluation subpackage provides **metrics for assessing LLM and pipeline outputs**: +LLM-as-judge metrics (faithfulness, relevancy, answer correctness, …) and deterministic +information-retrieval metrics (recall@k, nDCG, MRR, …). Every metric is a plain function +you call directly and combine however your harness needs — there is no gate, verdict, or +promotion machinery to opt into. + +--- + +## Installation + +The evaluation subpackage needs `numpy` for the embedding path and `ragas` (plus its +LangChain providers) for the RAGAS metrics. Install the optional extra: + +```bash +pip install "fireflyframework-agentic[evaluation]" +``` + +Everything except the RAGAS metrics works without `ragas` installed; the RAGAS functions +import it lazily and only fail if you call them without the extra. + +--- + +## Two metric families + +| Family | Module | Needs an LLM? | Use it to evaluate… | +|--------|--------|---------------|---------------------| +| **LLM-as-judge** | `evaluation.judge` | Most metrics yes (a few are deterministic/embedding) | The semantic quality of a model's answers and reports — faithfulness, relevancy, correctness, hallucination. | +| **Retrieval** | `evaluation.retrieval_metrics` | No (pure functions, no network) | The ranked retrieval that feeds the LLM — recall@k, precision@k, MRR, MAP, nDCG. | + +Both are re-exported from `fireflyframework_agentic.evaluation`. + +--- + +## LLM-as-judge metrics + +Each judge metric is an **async function** with the same signature: + +```python +async def metric(item: dict, ctx: EvalContext) -> dict | None +``` + +- `item` — a plain dict of the output under evaluation (see schema below). +- `ctx` — an `EvalContext` carrying the judge client and an optional embedder. +- The return is `{"score": float | None, …extra}`, or `None` when the metric cannot run + (e.g. an embedding metric with no embedder, or a missing field). See **Return shapes** below. + +### EvalContext and JudgeClient + +```python +from fireflyframework_agentic.evaluation import EvalContext, JudgeClient, build_embedder + +ctx = EvalContext( + client=JudgeClient("anthropic:claude-haiku-4-5"), + embedder=None, # optional framework embedder; required by semantic_recovery and RAGAS +) +``` + +`embedder` is any `fireflyframework_agentic` embedder. Build one from a +`":"` spec with `build_embedder` (openai, azure, cohere, google, +mistral, voyage, bedrock, ollama): + +```python +ctx = EvalContext( + client=JudgeClient("anthropic:claude-haiku-4-5"), + embedder=build_embedder("ollama:nomic-embed-text"), +) +``` + +The RAGAS metrics reuse this same framework embedder (wrapped for RAGAS), so the +evaluator embeds with the same provider as the rest of your pipeline. + +`JudgeClient` is an async multi-provider judge backed by the framework's `FireflyAgent` +(pydantic-ai). The model spec is `":"`, where provider is one of +`anthropic`, `openai`, `azure`, `ollama`. Each call returns a **validated, typed** Pydantic +model — the LLM's structured output is schema-checked rather than hand-parsed — and +`temperature` is pinned to `0.0` for stable verdicts. The provider reads its API key +(`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `AZURE_OPENAI_*`, `OLLAMA_HOST`) when the underlying +agent is first built, so constructing a `JudgeClient` never requires a secret. + +### Item schema + +The judge metrics read whichever keys they need and ignore the rest, so one `item` dict can +serve many metrics. + +**RAG / Q&A items** (single answer under test): + +```python +item = { + "question": "What is the boiling point of water at sea level?", + "answer": "Water boils at 100 degrees Celsius at sea level.", + "reference": "Water boils at 100 °C at standard atmospheric pressure.", + "contexts": ["...retrieved passage...", "..."], # used by RAGAS metrics +} +``` + +**Report / discovery items** (a structured pipeline output): + +```python +item = { + "findings": [{"id": ..., "title": ..., "description": ..., "severity": ..., + "evidence_refs": [{"evidence_id": ...}], ...}], + "evidence_index": [{"id": ..., "locator": "doc.md#L1", "excerpt": "..."}], + "process_graph": {"processes": [{"name": ..., "activities": [...], "decisions": [...]}]}, + "proposed_actions": [{"title": ..., "finding_id": ..., "expected_savings_fte": ...}], + "workspace": {"name": ..., "description": ...}, + "nc_items": [{"id": ..., "description": "a statement that is factually false"}], + "lexical_missed_ids": ["..."], # ids the lexical pass missed (semantic_recovery) + "champion": { ... another item ... }, # baseline for comparative_vs_champion +} +``` + +### Quick start — scoring Q&A pairs + +```python +import asyncio +from fireflyframework_agentic.evaluation import ( + EvalContext, JudgeClient, contains_answer, addresses_question, +) + +item = { + "question": "Who wrote Romeo and Juliet?", + "reference": "Romeo and Juliet was written by William Shakespeare around 1594–1596.", + "answer": "It was written by Shakespeare.", +} + +async def main(): + ctx = EvalContext(client=JudgeClient("anthropic:claude-haiku-4-5")) + contains = await contains_answer(item, ctx) # {"score": 0.0–1.0} or None + addresses = await addresses_question(item, ctx) # {"score": 0.0–1.0} or None + print(contains["score"], addresses["score"]) + +asyncio.run(main()) +``` + +See `examples/llm_eval_example.py` for a runnable version that scores a list of items +(built-in sample data or a JSONL file) and prints a table. + +### Metric catalog + +The three LLM-scored groups below are **all** "LLM-as-judge" — an LLM reads the output and +grades it. They differ in **who writes the rubric**: the *custom-rubric* groups use our own +hand-written prompts through `ctx.client` and return a schema we define, while the *RAGAS* +group runs the third-party `ragas` library's standardized, paper-backed metrics (which prompt +the LLM with their own internal logic and blend embeddings for some). There is deliberate +overlap — e.g. `faithfulness` (ours) vs. `ragas_faithfulness` (RAGAS) score the same idea two +ways. + +**Return shapes.** Every metric returns `{"score": float | None, …extra}` (or `None` when it +doesn't apply). Read `result["score"]` for the headline number — comparable across all metrics — +and the remaining keys for the breakdown (counts, ids, offending pairs). `score` is `None` only +where there is no natural [0, 1] aggregate: pure defect counts (`fabricated_entity`, +`contradiction`) and free-text / structured probes (`open_gap`, `comparative_vs_champion`). + +**Deterministic** — no LLM call, always available: + +| Metric | Returns | Measures | +|--------|---------|----------| +| `source_coverage` | `{score, cited, total, orphaned}` | Distinct source documents cited by ≥1 finding vs. all sources; `orphaned` lists uncited stems. | +| `excerpt_fill_rate` | `{score, populated, total}` | Fraction of `evidence_index` entries that carry a non-empty excerpt. | + +**Embedding** — requires `ctx.embedder`: + +| Metric | Returns | Measures | +|--------|---------|----------| +| `semantic_recovery` | `{score, lexical_recall, vector_recovered, tau, scored_denominator}` or `None` | Must-find recall: a lexical baseline (`lexical_recall`) lifted by a vector pass that recovers lexically-missed items above `tau`; `score` is the lexical-OR-vector union (hybrid recall). Returns `None` when no embedder is set. | + +**LLM-as-a-Judge (Custom rubric)** — our prompts via `ctx.client`: + +| Metric | Returns | Measures | +|--------|---------|----------| +| `faithfulness` | `{score, supported, total, unsupported_ids}` | Per-**finding** entailment against the finding's **own cited** excerpts. Cf. `ragas_faithfulness`. | +| `numeric_temporal_fidelity` | `{score, mismatches, count}` | Numbers/dates asserted in a finding that don't match its evidence. | +| `citation_relevance` | `{score, relevant, total}` | Context precision: fraction of cited passages actually relevant to the claim. | +| `nc_semantic_precision` | `{score, asserted, total, asserted_ids}` | How many negative-control falsehoods (`nc_items`) the output asserts or endorses. | +| `fabricated_entity` | `{score: None, count, entities}` | Systems/orgs/metrics named in the output but absent from the corpus. | +| `contradiction` | `{score: None, count, pairs}` | Internally contradictory finding pairs. | +| `open_gap` | `{score: None, gap}` | G-Eval open probe: the most important issue the output missed (free-text, no score). | +| `actionability` | `{score, rated}` | Average 0–1 rating of whether proposed actions are specific, quantified, and linked. | +| `severity_calibration` | `{score, miscalibrated, total, verdicts}` | Whether each finding's stated severity matches its evidence (under/over/calibrated). | +| `answer_relevancy` | `{score}` | Does the output address the stated workspace intention? | +| `surface_deduplication` | `{score, distinct, redundant, total, redundant_pairs}` | Fraction of near-duplicate process-graph nodes that are genuinely distinct. | +| `comparative_vs_champion` | `{score: None, candidate, champion, more_consistent}` or `None` | Pairwise five-axis review of candidate vs. `item["champion"]`. `None` if no champion. | + +**LLM-as-a-Judge (RAGAS)** — the `ragas` package's own LLM judges; needs the `ragas` extra and `ctx.client` (plus an embedder for some): + +| Metric | Returns | Measures | +|--------|---------|----------| +| `answer_correctness` | `{score}` or `None` | Semantic F1 of the answer against the reference. | +| `ragas_faithfulness` | `{score}` or `None` | Per-**claim** grounding: fraction of the answer's atomic claims inferable from the retrieved `contexts`. Cf. `faithfulness`. | +| `context_recall` | `{score}` or `None` | Reference coverage by the retrieved `contexts`. | +| `context_precision` | `{score}` or `None` | Retrieved `contexts` relevant to the question. | + +**LLM-as-a-Judge (Custom rubric — RAG Q&A)** — our prompts via `ctx.client`: + +| Metric | Returns | Measures | +|--------|---------|----------| +| `contains_answer` | `{score}` or `None` | Does the answer contain the correct information from the reference? | +| `addresses_question` | `{score}` or `None` | Does the answer directly address what the question asks? | + +#### `faithfulness` vs. `ragas_faithfulness` + +Both check grounding (and both return `{score, …}`). `faithfulness` judges each **finding** +against its own **cited** excerpts — `score = supported / total`, plus `unsupported_ids`; +`ragas_faithfulness` decomposes a RAG **answer** into atomic claims and its `score` is the +fraction inferable from the retrieved **contexts**. + +### Running a metric family at once + +`run_judge()` runs a selected family of metrics concurrently and collects them into an +`AdvisoryReport`. The `metrics` argument picks the family: + +- `"basic"` — domain-agnostic LLM/RAG answer-quality (`BASIC_METRICS`): `contains_answer`, + `addresses_question`, and the four RAGAS metrics. +- `"process_mining"` — flyradar discovery-report (`PROCESS_MINING_METRICS`): the 15 metrics + that read `findings`/`evidence_index`/`process_graph`/… +- `"all"` (default) — both families. + +It is best-effort and never raises — any metric that fails is recorded in `report.errors` +instead of propagating. + +```python +import asyncio +from fireflyframework_agentic.evaluation import run_judge, EvalContext, JudgeClient + +async def main(): + ctx = EvalContext(client=JudgeClient("anthropic:claude-haiku-4-5")) + report = await run_judge(item, ctx, metrics="basic", pipeline_model="anthropic:claude-sonnet-4-6") + print(report.metrics) # {metric_name: result, ...} + print(report.errors) # ["metric: ExceptionType: message", ...] + +asyncio.run(main()) +``` + +`AdvisoryReport` fields: + +| Field | Type | Description | +|-------|------|-------------| +| `judge_model` | `str` | The judge model spec used. | +| `same_provider_caveat` | `bool` | `True` when the judge and the evaluated pipeline share a provider (self-grading risk). | +| `metrics` | `dict` | Per-metric results, keyed by metric name. | +| `errors` | `list[str]` | Per-metric failures captured best-effort. | + +--- + +## Retrieval metrics + +Deterministic IR metrics over ranked retrieval results — no LLM and no network, the same +design as scikit-learn or MS MARCO evaluation scripts. Each is a plain function over a list +of result rows. + +### Row schema + +```python +results = [ + { + "retrieved": [{"rank": 1, "source_id": "SOP-002.md", "is_gold": True}, + {"rank": 2, "source_id": "SOP-001.md", "is_gold": False}], + "gold": ["SOP-002.md"], # gold source identifiers + # optional: + "no_answer": False, # model refused / produced no answer + "answer": "...", # used for no_answer detection if no_answer absent + "citations": [{"is_gold": True}], + }, +] +``` + +`rank` is 1-based (rank 1 is the top hit). Duplicate sources are de-duplicated by +`source_id`, keeping the best-ranked chunk. + +### Metric catalog + +| Function | Signature | Measures | +|----------|-----------|----------| +| `hit_at_k` | `(results, k) -> float` | Fraction of queries with ≥1 gold document in top-k. | +| `recall_at_k` | `(results, k) -> float` | Mean fraction of gold documents found in top-k. | +| `precision_at_k` | `(results, k) -> float` | Mean fraction of top-k results that are gold. | +| `mrr` | `(results, k=10) -> float` | Mean reciprocal rank of the first gold hit. | +| `map_score` | `(results, k=10) -> float` | Mean average precision at k. | +| `ndcg` | `(results, k=10) -> float` | Mean normalised discounted cumulative gain at k. | +| `no_answer_rate` | `(results) -> float \| None` | Fraction of queries with no answer. `None` if no results. | +| `citation_precision` | `(results) -> float \| None` | Precision of in-answer citations vs. the gold set. `None` if no citations. | + +### Example + +```python +from fireflyframework_agentic.evaluation import recall_at_k, ndcg, mrr + +print(f"Recall@5: {recall_at_k(results, 5):.3f}") +print(f"nDCG@10: {ndcg(results):.3f}") +print(f"MRR@10: {mrr(results):.3f}") +``` + +--- + +## Reference + +All symbols below are importable from `fireflyframework_agentic.evaluation`. + +### Core types + +| Symbol | Kind | Description | +|--------|------|-------------| +| `EvalContext` | Pydantic model | Carries `client` and an optional `embedder` for the judge metrics. | +| `BASIC_METRICS` / `PROCESS_MINING_METRICS` | `tuple[str, ...]` | Metric-name families selectable via `run_judge(..., metrics=...)`. | +| `build_embedder` | Function | Build a framework embedder from a `":"` spec (openai/azure/cohere/google/mistral/voyage/bedrock/ollama). | +| `JudgeClient` | Class | Async multi-provider (`anthropic`/`openai`/`azure`/`ollama`) judge backed by `FireflyAgent`; returns validated typed output. | +| `AdvisoryReport` | Pydantic model | Aggregated `run_judge` output: `metrics`, `errors`, and run metadata. | +| `Metric` | Type alias | `Callable[[dict, EvalContext], Awaitable[dict \| float \| None]]`. | +| `parse_model` | Function | Split `"provider:model"` into `(provider, model)`. | +| `same_provider` | Function | `True` if two model specs share a known provider prefix. | + +### Judge metrics + +`source_coverage`, `excerpt_fill_rate`, `semantic_recovery`, `faithfulness`, +`numeric_temporal_fidelity`, `citation_relevance`, `nc_semantic_precision`, +`fabricated_entity`, `contradiction`, `open_gap`, `actionability`, +`severity_calibration`, `answer_relevancy`, `surface_deduplication`, +`comparative_vs_champion`, `contains_answer`, `addresses_question`, +`answer_correctness`, `ragas_faithfulness`, `context_recall`, `context_precision`, +and the orchestrator `run_judge`. + +### Retrieval metrics + +`hit_at_k`, `recall_at_k`, `precision_at_k`, `mrr`, `map_score`, `ndcg`, +`no_answer_rate`, `citation_precision`. diff --git a/examples/llm_eval_example.py b/examples/llm_eval_example.py new file mode 100644 index 00000000..145e62ab --- /dev/null +++ b/examples/llm_eval_example.py @@ -0,0 +1,119 @@ +# Copyright 2026 Firefly Software Foundation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""LLM-as-judge evaluation example. + +Score a set of Q&A pairs using the evaluation metrics: + - contains_answer — does the answer contain the correct information? + - addresses_question — does the answer directly address what was asked? + +Usage:: + + python examples/llm_eval_example.py --model anthropic:claude-haiku-4-5 + + # Or score from a JSONL file instead of the built-in sample data: + python examples/llm_eval_example.py \\ + --model anthropic:claude-haiku-4-5 \\ + --items-file items.jsonl + +Items JSONL format — one JSON object per line:: + + {"question": "...", "answer": "...", "reference": "..."} +""" + +from __future__ import annotations + +import argparse +import asyncio +import json +from pathlib import Path + +from fireflyframework_agentic.evaluation import ( + EvalContext, + JudgeClient, + addresses_question, + contains_answer, +) + +# Sample data used when no --items-file is provided. +_SAMPLE_ITEMS = [ + { + "question": "What is the boiling point of water at sea level?", + "reference": "Water boils at 100 °C at standard atmospheric pressure.", + "answer": "Water boils at 100 degrees Celsius at sea level.", + }, + { + "question": "Who wrote Romeo and Juliet?", + "reference": "Romeo and Juliet was written by William Shakespeare around 1594–1596.", + "answer": "It was written by Shakespeare.", + }, + { + "question": "What is the capital of France?", + "reference": "The capital of France is Paris.", + "answer": "The weather in France is generally mild.", + }, +] + + +async def score_items(items: list[dict], ctx: EvalContext) -> list[dict]: + tasks = [(contains_answer(item, ctx), addresses_question(item, ctx)) for item in items] + pairs = await asyncio.gather(*[asyncio.gather(ca, aq) for ca, aq in tasks]) + return [ + { + "question": item["question"], + "contains_answer": ca["score"] if ca else None, + "addresses_question": aq["score"] if aq else None, + } + for item, (ca, aq) in zip(items, pairs, strict=True) + ] + + +async def main(args: argparse.Namespace) -> None: + if args.items_file: + lines = Path(args.items_file).read_text(encoding="utf-8").strip().splitlines() + items = [json.loads(line) for line in lines if line.strip()] + else: + items = _SAMPLE_ITEMS + + ctx = EvalContext(client=JudgeClient(args.model)) + results = await score_items(items, ctx) + + print(f"\n{'Question':<45} {'contains':>8} {'addresses':>9}") + print("-" * 63) + for r in results: + q = r["question"][:43] + ".." if len(r["question"]) > 45 else r["question"] + ca = f"{r['contains_answer']:.2f}" if r["contains_answer"] is not None else " n/a" + aq = f"{r['addresses_question']:.2f}" if r["addresses_question"] is not None else " n/a" + print(f"{q:<45} {ca:>8} {aq:>9}") + + scored = [r for r in results if r["contains_answer"] is not None] + if scored: + avg_ca = sum(r["contains_answer"] for r in scored) / len(scored) + avg_aq = sum(r["addresses_question"] for r in scored) / len(scored) + print("-" * 63) + print(f"{'Average':<45} {avg_ca:>8.2f} {avg_aq:>9.2f}") + print(f"\n{len(items)} items scored.") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description="Score Q&A pairs with LLM-as-judge metrics.") + parser.add_argument( + "--model", + default="anthropic:claude-haiku-4-5", + help="Judge model spec (provider:model).", + ) + parser.add_argument( + "--items-file", default=None, help="Optional JSONL file of {question, answer, reference} items." + ) + asyncio.run(main(parser.parse_args())) diff --git a/fireflyframework_agentic/evaluation/__init__.py b/fireflyframework_agentic/evaluation/__init__.py new file mode 100644 index 00000000..186160b1 --- /dev/null +++ b/fireflyframework_agentic/evaluation/__init__.py @@ -0,0 +1,91 @@ +"""Evaluation metrics for LLM and pipeline outputs. + +LLM-as-judge metrics (``judge``), the spec-driven embedder factory (``embedder``), +and deterministic retrieval metrics (``retrieval_metrics``). +""" + +from fireflyframework_agentic.evaluation.embedder import build_embedder +from fireflyframework_agentic.evaluation.judge import ( + BASIC_METRICS, + PROCESS_MINING_METRICS, + AdvisoryReport, + EvalContext, + JudgeClient, + Metric, + actionability, + addresses_question, + answer_correctness, + answer_relevancy, + citation_relevance, + comparative_vs_champion, + contains_answer, + context_precision, + context_recall, + contradiction, + excerpt_fill_rate, + fabricated_entity, + faithfulness, + nc_semantic_precision, + numeric_temporal_fidelity, + open_gap, + parse_model, + ragas_faithfulness, + run_judge, + same_provider, + semantic_recovery, + severity_calibration, + source_coverage, + surface_deduplication, +) +from fireflyframework_agentic.evaluation.retrieval_metrics import ( + citation_precision, + hit_at_k, + map_score, + mrr, + ndcg, + no_answer_rate, + precision_at_k, + recall_at_k, +) + +__all__ = [ + "BASIC_METRICS", + "PROCESS_MINING_METRICS", + "AdvisoryReport", + "EvalContext", + "JudgeClient", + "Metric", + "actionability", + "addresses_question", + "answer_correctness", + "answer_relevancy", + "build_embedder", + "citation_precision", + "citation_relevance", + "comparative_vs_champion", + "contains_answer", + "context_precision", + "context_recall", + "contradiction", + "excerpt_fill_rate", + "fabricated_entity", + "faithfulness", + "hit_at_k", + "map_score", + "mrr", + "nc_semantic_precision", + "ndcg", + "no_answer_rate", + "numeric_temporal_fidelity", + "open_gap", + "parse_model", + "precision_at_k", + "ragas_faithfulness", + "recall_at_k", + "run_judge", + "same_provider", + "semantic_recovery", + "severity_calibration", + "source_coverage", + "surface_deduplication", +] diff --git a/fireflyframework_agentic/evaluation/embedder.py b/fireflyframework_agentic/evaluation/embedder.py new file mode 100644 index 00000000..7fe82ac1 --- /dev/null +++ b/fireflyframework_agentic/evaluation/embedder.py @@ -0,0 +1,82 @@ +# Copyright 2026 Firefly Software Foundation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Resolve a ``:`` spec to a framework embedder. + +Mirrors flycanon's ``embedding_service._build_embedder``: one branch per +provider shipped by ``fireflyframework_agentic.embeddings``. Per-provider +imports are deferred so a spec that never touches a given provider doesn't +require its SDK to be installed. +""" + +from __future__ import annotations + +import os + +from fireflyframework_agentic.embeddings.base import BaseEmbedder + + +def build_embedder(spec: str, *, dimensions: int | None = None, batch_size: int = 64) -> BaseEmbedder: + """Build a framework embedder from a ``":"`` spec. + + Supported providers: openai, azure, cohere, google, mistral, voyage, + bedrock, ollama. Raises ``ValueError`` on a malformed spec or unknown + provider. + """ + if ":" not in spec: + raise ValueError(f"embedder spec must be ':' (got {spec!r})") + provider, _, model = spec.partition(":") + p = provider.strip().lower() + if p == "openai": + from fireflyframework_agentic.embeddings.providers.openai import OpenAIEmbedder # noqa: PLC0415 + + return OpenAIEmbedder(model=model, dimensions=dimensions, batch_size=batch_size) + if p in ("azure", "azure-openai"): + from fireflyframework_agentic.embeddings.providers.azure import AzureEmbedder # noqa: PLC0415 + + return AzureEmbedder( + model=model, + dimensions=dimensions, + batch_size=batch_size, + azure_endpoint=os.environ.get("AZURE_OPENAI_ENDPOINT", ""), + ) + if p == "cohere": + from fireflyframework_agentic.embeddings.providers.cohere import CohereEmbedder # noqa: PLC0415 + + return CohereEmbedder(model=model, dimensions=dimensions, batch_size=batch_size) + if p in ("google", "gemini"): + from fireflyframework_agentic.embeddings.providers.google import GoogleEmbedder # noqa: PLC0415 + + return GoogleEmbedder(model=model, dimensions=dimensions, batch_size=batch_size) + if p == "mistral": + from fireflyframework_agentic.embeddings.providers.mistral import MistralEmbedder # noqa: PLC0415 + + return MistralEmbedder(model=model, dimensions=dimensions, batch_size=batch_size) + if p == "voyage": + from fireflyframework_agentic.embeddings.providers.voyage import VoyageEmbedder # noqa: PLC0415 + + return VoyageEmbedder(model=model, dimensions=dimensions, batch_size=batch_size) + if p == "bedrock": + from fireflyframework_agentic.embeddings.providers.bedrock import BedrockEmbedder # noqa: PLC0415 + + return BedrockEmbedder(model=model, dimensions=dimensions, batch_size=batch_size) + if p == "ollama": + from fireflyframework_agentic.embeddings.providers.ollama import OllamaEmbedder # noqa: PLC0415 + + base_url = os.environ.get("OLLAMA_HOST", "http://localhost:11434") + return OllamaEmbedder(model=model, dimensions=dimensions, base_url=base_url, batch_size=batch_size) + raise ValueError( + f"unknown embedding provider {provider!r}; supported: " + "openai, azure, cohere, google, mistral, voyage, bedrock, ollama" + ) diff --git a/fireflyframework_agentic/evaluation/judge.py b/fireflyframework_agentic/evaluation/judge.py new file mode 100644 index 00000000..3c14cfdc --- /dev/null +++ b/fireflyframework_agentic/evaluation/judge.py @@ -0,0 +1,1038 @@ +"""Evaluation judge — async metrics for flyradar and flycanon pipelines. + +Every metric: async def metric_name(item: dict, ctx: EvalContext) -> dict | None +Each result is {"score": float | None, **extra} — read result["score"] for the headline. + +Flyradar item keys: findings, evidence_index, process_graph, proposed_actions, + workspace, reports, lexical_missed_ids, nc_items, champion +Flycanon item keys: question, answer, reference, contexts +""" + +from __future__ import annotations + +import asyncio +import math +import os +from collections.abc import Awaitable, Callable + +from pydantic import BaseModel, ConfigDict + +from fireflyframework_agentic.agents import FireflyAgent +from fireflyframework_agentic.embeddings.base import BaseEmbedder +from fireflyframework_agentic.embeddings.similarity import cosine_similarity + +# ── judge client ───────────────────────────────────────────────────────────────── + +_AGENT_NAME = "evaluation-judge" + + +def parse_model(spec: str) -> tuple[str, str]: + """Split "provider:model" -> (provider, model). Bare spec -> ("unknown", spec).""" + spec = (spec or "").strip() + if ":" not in spec: + return "unknown", spec + provider, model = spec.split(":", 1) + return provider.strip().lower(), model.strip() + + +def same_provider(pipeline_model: str, judge_model: str) -> bool: + """True iff both specs share the same known provider prefix.""" + p, _ = parse_model(pipeline_model) + j, _ = parse_model(judge_model) + if p == "unknown" or j == "unknown": + return False + return p == j + + +class JudgeClient: + """Async multi-provider judge backed by :class:`FireflyAgent`. + + Each ``judge`` call returns a validated instance of the requested pydantic + ``output_type`` — schema enforcement replaces hand-rolled JSON parsing. + ``temperature`` is pinned to 0.0 for deterministic verdicts. Agents are built + lazily and cached per ``(system, output_type, max_tokens)``; transient + rate-limit / 5xx errors and output-validation failures are retried by + FireflyAgent / pydantic-ai (``max_retries``). The provider reads its API key + when the agent is first built, so constructing a client never needs a secret. + """ + + def __init__(self, model: str, timeout: int = 120, max_retries: int = 3) -> None: + self.model_spec = model + self.provider, self.model = parse_model(model) + self.timeout = timeout + self.max_retries = max_retries + self._agents: dict[tuple[str, type, int], FireflyAgent] = {} + + def _agent[T: BaseModel](self, system: str, output_type: type[T], max_tokens: int) -> FireflyAgent: + key = (system, output_type, max_tokens) + agent = self._agents.get(key) + if agent is None: + agent = FireflyAgent( + name=_AGENT_NAME, + model=self.model_spec, + instructions=system, + output_type=output_type, + model_settings={"temperature": 0.0, "max_tokens": max_tokens}, + retries=self.max_retries, + auto_register=False, + ) + self._agents[key] = agent + return agent + + async def judge[T: BaseModel](self, system: str, user: str, output_type: type[T], max_tokens: int = 1024) -> T: + """Send (system, user) to the model and return a validated ``output_type``. + + Raises on exhausted retries / unknown provider / output that cannot be + coerced to ``output_type`` — callers must not treat a failure as a verdict. + """ + agent = self._agent(system, output_type, max_tokens) + result = await agent.run(user, timeout=self.timeout) + return result.output + + +Metric = Callable[["dict", "EvalContext"], Awaitable["dict | None"]] + +SYSTEM = "You are a meticulous evaluator of a process-mining discovery report. Return ONLY a JSON object." + +SYSTEM_RAG = "You are an evaluator of a RAG system's answers. Return ONLY a JSON object." + +RUBRIC = ( + "Score the ANSWER on two metrics:\n" + "- contains_answer (0.0-1.0): Does the answer contain the correct information from the REFERENCE?\n" + "- addresses_question (0.0-1.0): Does the answer directly address what the QUESTION is asking?\n" + 'Reply with ONLY {"contains_answer": , "addresses_question": }.' +) + +# ── structured judge outputs (validated by the model via FireflyAgent) ─────────── + + +class _Verdict(BaseModel): + verdict: str = "" + reason: str = "" + + +class _Mismatch(BaseModel): + value: str = "" + source: str = "" + + +class _Mismatches(BaseModel): + mismatches: list[_Mismatch] = [] + + +class _Relevant(BaseModel): + relevant: str = "" + + +class _Asserted(BaseModel): + asserted: str = "" + + +class _Fabricated(BaseModel): + fabricated: list[str] = [] + + +class _Pairs(BaseModel): + pairs: list[list[str]] = [] + + +class _Gap(BaseModel): + gap: str = "" + + +class _Score(BaseModel): + score: float | None = None + + +class _Calibration(BaseModel): + calibration: str = "calibrated" + + +class _Comparison(BaseModel): + candidate: dict = {} + champion: dict = {} + more_consistent: str = "" + + +class _RagScore(BaseModel): + contains_answer: float | None = None + addresses_question: float | None = None + + +class EvalContext(BaseModel): + model_config = ConfigDict(arbitrary_types_allowed=True) + + client: JudgeClient + embedder: BaseEmbedder | None = None + + +class AdvisoryReport(BaseModel): + """Aggregated output of :func:`run_judge`: a plain metrics bag. + + metrics maps metric-name -> the per-metric result (a small dict or float). + errors lists per-metric failures captured by run_judge's best-effort + try/except so nothing propagates. judge_model is run metadata; + same_provider_caveat flags self-grading risk (the judge shares the evaluated + pipeline's provider). + """ + + judge_model: str + same_provider_caveat: bool + metrics: dict = {} + errors: list[str] = [] + + +# ── shared accessors ─────────────────────────────────────────────────────────── + + +def _evidence_index(item: dict) -> dict[str, dict]: + return {ev.get("id"): ev for ev in item.get("evidence_index", []) if ev.get("id")} + + +def _cited_excerpts(finding: dict, evidence_index: dict[str, dict]) -> list[str]: + """Excerpts of the evidence a finding cites (via evidence_refs.evidence_id).""" + out: list[str] = [] + for ref in finding.get("evidence_refs", []): + ev = evidence_index.get(ref.get("evidence_id", "")) + if ev: + excerpt = ev.get("excerpt") or "" + if excerpt: + out.append(excerpt) + return out + + +def _output_text(item: dict) -> str: + """All free text the model emitted: finding titles+descriptions + reports.""" + parts: list[str] = [] + for f in item.get("findings", []): + parts.append(f.get("title", "")) + parts.append(f.get("description", "")) + for r in item.get("reports", []): + parts.append(str(r)) + return "\n".join(p for p in parts if p) + + +def _workspace_intention(item: dict) -> str: + ws = item.get("workspace") or {} + return f"{ws.get('name', '')}\n{ws.get('description', '')}".strip() + + +def _source_stem(locator: str) -> str: + """Return the part before the first '#', or the full string if no '#'.""" + idx = locator.find("#") + return locator[:idx] if idx != -1 else locator + + +async def _judge_all[T: BaseModel](ctx: EvalContext, system: str, users: list[str], output_type: type[T]) -> list[T]: + """Judge a list of user prompts concurrently against one system prompt. + + Failures propagate (no swallowing into a fake verdict): a failed call raises, + so run_judge records it in report.errors instead of scoring it as a result. + """ + return list(await asyncio.gather(*[ctx.client.judge(system, u, output_type) for u in users])) + + +def _scored(score: float | None, **extra: object) -> dict: + """Uniform metric result: a leading ``score`` float (or None) then structured extras. + + Every metric returns this shape so results compare apples-to-apples — read + ``result["score"]`` for the headline number and the remaining keys for the breakdown. + ``score`` is None for metrics with no natural [0, 1] aggregate (pure defect counts and + free-text probes). + """ + return {"score": score, **extra} + + +# ── [D] DETERMINISTIC — no LLM, always available ──────────────────────────────── + + +async def source_coverage(item: dict, ctx: EvalContext) -> dict: # noqa: ARG001 + """Distinct source documents cited by >=1 finding vs all source documents. + + Returns {cited, total, orphaned} where orphaned is the sorted list of + source stems present in evidence_index but cited by no finding. + """ + ev_idx = _evidence_index(item) + all_stems = {_source_stem(ev.get("locator", "")) for ev in item.get("evidence_index", []) if ev.get("locator")} + cited_stems: set[str] = set() + for f in item.get("findings", []): + for ref in f.get("evidence_refs", []): + ev = ev_idx.get(ref.get("evidence_id", "")) + if ev and ev.get("locator"): + cited_stems.add(_source_stem(ev["locator"])) + cited_stems &= all_stems + orphaned = sorted(all_stems - cited_stems) + return _scored( + round(len(cited_stems) / len(all_stems), 4) if all_stems else None, + cited=len(cited_stems), + total=len(all_stems), + orphaned=orphaned, + ) + + +async def excerpt_fill_rate(item: dict, ctx: EvalContext) -> dict: # noqa: ARG001 + """Fraction of evidence_index entries with a non-empty excerpt. + + Returns {populated, total}. + """ + entries = item.get("evidence_index", []) + populated = sum(1 for ev in entries if (ev.get("excerpt") or "").strip()) + return _scored(round(populated / len(entries), 4) if entries else None, populated=populated, total=len(entries)) + + +# ── [E] EMBEDDING — needs embedder ─────────────────────────────────────────────── + + +async def semantic_recovery(item: dict, ctx: EvalContext, tau: float = 0.70) -> dict | None: + """Must-find recall: a lexical baseline lifted by a vector pass (the hybrid). + + Upstream, the eval harness matches the expected ("must-find") items against the + discovery output by surface/keyword (lexical) matching; the ids it could NOT match + arrive here as ``item["lexical_missed_ids"]``. This metric runs a second, *vector* + pass over those misses: it embeds each missed item and the candidate finding texts + and counts it recovered when the best cosine similarity is >= ``tau``. The three + recall views make the lexical -> vector -> hybrid progression explicit: + + - ``score`` : hybrid recall — lexical hits PLUS vector recoveries. + - ``lexical_recall`` : lexical-only baseline. + - ``vector_recovered`` : the misses the vector pass recovered, each with its cosine. + + Returns None when ``ctx.embedder`` is unset (the vector pass needs embeddings). + """ + if ctx.embedder is None: + return None + + missed = set(item.get("lexical_missed_ids", []) or []) + + # Candidate surface the vector pass scores against: finding descriptions plus + # their cited evidence excerpts. + ev_idx = _evidence_index(item) + candidate_texts: list[str] = [] + for f in item.get("findings", []): + desc = f.get("description", "") + if desc: + candidate_texts.append(desc) + candidate_texts.extend(_cited_excerpts(f, ev_idx)) + + all_findings = item.get("findings", []) + denom = max(len(all_findings), 1) + lexical_hits = sum(1 for f in all_findings if f.get("id") not in missed) + + # The lexical misses we can embed (those carrying a description). + missed_descs: list[tuple[str, str]] = [ + (f.get("id", ""), f.get("description", "")) + for f in all_findings + if f.get("id") in missed and f.get("description") + ] + + if not missed_descs or not candidate_texts: + lexical_recall = round(lexical_hits / denom, 4) + return _scored( + lexical_recall, + lexical_recall=lexical_recall, + vector_recovered=[], + tau=tau, + scored_denominator=denom, + ) + + item_vecs = await ctx.embedder._embed_batch([desc for _fid, desc in missed_descs]) + cand_vecs = await ctx.embedder._embed_batch(candidate_texts) + + vector_recovered: list[dict] = [] + for (fid, _desc), ivec in zip(missed_descs, item_vecs, strict=False): + best = max((cosine_similarity(ivec, cvec) for cvec in cand_vecs), default=0.0) + if best >= tau: + vector_recovered.append({"id": fid, "cosine": round(best, 4)}) + + return _scored( + round((lexical_hits + len(vector_recovered)) / denom, 4), + lexical_recall=round(lexical_hits / denom, 4), + vector_recovered=vector_recovered, + tau=tau, + scored_denominator=denom, + ) + + +# ── [J] JUDGE — needs ctx.client.judge(system, user, output_type) ──────────────── + + +async def faithfulness(item: dict, ctx: EvalContext) -> dict: + """Per-finding entailment: does each finding's CITED evidence support its claim? + + Custom-rubric judge over a discovery report. For every finding it gathers the excerpts + the finding explicitly cites (evidence_refs -> evidence_index) and asks the LLM a single + binary verdict — does that cited evidence ENTAIL the finding's claim (SUPPORTED / + NOT_SUPPORTED)? A finding that cites no evidence is counted unsupported without an LLM + call. The unit judged is the whole finding, compared only against its own citations. + + Returns {score, supported, total, unsupported_ids}; score = supported / total. + + Differs from ``ragas_faithfulness`` (which measures the same hallucination concept): + that one works on a RAG answer — it decomposes the ANSWER into atomic claims and scores + the fraction inferable from the retrieved ``contexts`` (not citations), returning a + single float in [0, 1]. Use this one for per-finding SUPPORTED/NOT_SUPPORTED + accountability when findings carry explicit citations. + """ + ev_idx = _evidence_index(item) + findings = item.get("findings", []) + cited = [(f, _cited_excerpts(f, ev_idx)) for f in findings] + users = [ + "Does the cited evidence span ENTAIL the claim made in this finding?\n" + 'Reply with ONLY {"verdict": "SUPPORTED" or "NOT_SUPPORTED", "reason": ""}.\n\n' + f"FINDING: {f.get('description', '')}\n" + f"CITED EVIDENCE: {' || '.join(excerpts)}" + for f, excerpts in cited + if excerpts + ] + answers = iter(await _judge_all(ctx, SYSTEM, users, _Verdict)) + supported = 0 + unsupported_ids: list[str] = [] + for f, excerpts in cited: + fid = f.get("id", "?") + if not excerpts: + unsupported_ids.append(fid) + continue + if str(next(answers).verdict).upper() == "SUPPORTED": + supported += 1 + else: + unsupported_ids.append(fid) + return _scored( + round(supported / len(findings), 4) if findings else None, + supported=supported, + total=len(findings), + unsupported_ids=unsupported_ids, + ) + + +async def numeric_temporal_fidelity(item: dict, ctx: EvalContext) -> dict: + """Flag numbers/dates asserted in a finding that do NOT match its evidence. + + Returns {score, mismatches: [{finding_id, value, source}], count}; score is the fraction + of evidence-cited findings with no numeric/temporal mismatch (None if none were scored). + """ + ev_idx = _evidence_index(item) + scored = [(f, excerpts) for f in item.get("findings", []) if (excerpts := _cited_excerpts(f, ev_idx))] + users = [ + "List every specific number or date asserted in the FINDING that does " + "NOT match the CITED EVIDENCE.\n" + 'Reply with ONLY {"mismatches": [{"value": "", "source": ""}]}. ' + "Empty list if all match.\n\n" + f"FINDING: {f.get('description', '')}\n" + f"CITED EVIDENCE: {' || '.join(excerpts)}" + for f, excerpts in scored + ] + answers = await _judge_all(ctx, SYSTEM, users, _Mismatches) + mismatches: list[dict] = [] + for (f, _excerpts), answer in zip(scored, answers, strict=False): + for m in answer.mismatches: + mismatches.append( + { + "finding_id": f.get("id", "?"), + "value": m.value, + "source": m.source, + } + ) + bad = len({m["finding_id"] for m in mismatches}) + return _scored( + round(1 - bad / len(scored), 4) if scored else None, + mismatches=mismatches, + count=len(mismatches), + ) + + +async def citation_relevance(item: dict, ctx: EvalContext) -> dict: + """Context precision: fraction of cited passages actually relevant to the claim. + + Returns {score, relevant, total}; score is that precision (relevant / total). + """ + ev_idx = _evidence_index(item) + users: list[str] = [] + for f in item.get("findings", []): + desc = f.get("description", "") + for ref in f.get("evidence_refs", []): + ev = ev_idx.get(ref.get("evidence_id", "")) + if not ev: + continue + excerpt = ev.get("excerpt") or "" + if not excerpt: + continue + users.append( + "Is this cited passage actually relevant to / used by this claim?\n" + 'Reply with ONLY {"relevant": "yes" or "no"}.\n\n' + f"CLAIM: {desc}\n" + f"CITED PASSAGE: {excerpt}" + ) + answers = await _judge_all(ctx, SYSTEM, users, _Relevant) + total = len(users) + relevant = sum(1 for a in answers if str(a.relevant).lower() == "yes") + return _scored(round(relevant / total, 4) if total else None, relevant=relevant, total=total) + + +async def nc_semantic_precision(item: dict, ctx: EvalContext) -> dict: + """Count negative-control falsehoods the output asserts or endorses. + + Reads item["nc_items"] as list of {"id": ..., "description": ...} dicts. + Returns {score, asserted, total, asserted_ids}; score is the fraction of negative + controls NOT endorsed (1 - asserted / total) — higher is better. + """ + output_text = _output_text(item) + nc_items: list[dict] = item.get("nc_items", []) + users = [ + "Does the OUTPUT assert or endorse the following FALSE statement?\n" + 'Reply with ONLY {"asserted": "yes" or "no"}.\n\n' + f"FALSE STATEMENT: {nc.get('description', '')}\n" + f"OUTPUT:\n{output_text}" + for nc in nc_items + ] + answers = await _judge_all(ctx, SYSTEM, users, _Asserted) + asserted_ids = [ + nc.get("id", "?") for nc, a in zip(nc_items, answers, strict=False) if str(a.asserted).lower() == "yes" + ] + return _scored( + round(1 - len(asserted_ids) / len(nc_items), 4) if nc_items else None, + asserted=len(asserted_ids), + total=len(nc_items), + asserted_ids=asserted_ids, + ) + + +async def fabricated_entity(item: dict, ctx: EvalContext) -> dict: + """Count systems/orgs/metrics named in the output but absent from the corpus. + + Returns {score, count, entities}; score is None (a pure defect count has no denominator). + """ + output_text = _output_text(item) + corpus = "\n".join(f"{ev.get('locator', '')} :: {ev.get('excerpt', '')}" for ev in item.get("evidence_index", [])) + user = ( + "List any system, organization, or metric NAMED in the OUTPUT that does NOT " + "appear anywhere in the CORPUS EVIDENCE.\n" + 'Reply with ONLY {"fabricated": ["", ...]}. Empty list if none.\n\n' + f"OUTPUT:\n{output_text}\n\n" + f"CORPUS EVIDENCE:\n{corpus}" + ) + answer = await ctx.client.judge(SYSTEM, user, _Fabricated) + entities = answer.fabricated + return _scored(None, count=len(entities), entities=list(entities)) + + +async def contradiction(item: dict, ctx: EvalContext) -> dict: + """Count internally contradictory finding pairs. + + Returns {score, count, pairs}; score is None (a pure defect count has no denominator). + """ + lines = [] + for f in item.get("findings", []): + lines.append(f"{f.get('id', '?')}: {f.get('title', '')} — {f.get('description', '')}") + user = ( + "Are any two of these FINDINGS mutually contradictory? List each contradicting pair.\n" + 'Reply with ONLY {"pairs": [["", ""], ...]}. Empty list if none.\n\n' + "\n".join(lines) + ) + answer = await ctx.client.judge(SYSTEM, user, _Pairs) + pairs = answer.pairs + return _scored(None, count=len(pairs), pairs=[list(p) for p in pairs]) + + +async def open_gap(item: dict, ctx: EvalContext) -> dict: + """G-Eval open probe: the most important process issue the output missed. + + Returns {score, gap}; score is None (free-text advisory narrative, no score). + """ + pg = item.get("process_graph") or {} + pg_summary = f"process_graph has {len(pg.get('processes', []))} processes" + user = ( + "Given this corpus scope and output, what important process issue did the " + "output FAIL to surface?\n" + 'Reply with ONLY {"gap": ""}.\n\n' + f"WORKSPACE SCOPE: {_workspace_intention(item)}\n" + f"{pg_summary}\n" + f"OUTPUT:\n{_output_text(item)}" + ) + answer = await ctx.client.judge(SYSTEM, user, _Gap) + return _scored(None, gap=str(answer.gap)) + + +async def actionability(item: dict, ctx: EvalContext) -> dict: + """Average 0-1 rating of whether proposed actions are specific+quantified+linked. + + Returns {score, rated}. + """ + actions = item.get("proposed_actions", []) or [] + finding_ids = {f.get("id") for f in item.get("findings", [])} + users = [ + "Rate whether this proposed action is SPECIFIC, QUANTIFIED, and LINKED to a " + "finding.\n" + 'Reply with ONLY {"score": }.\n\n' + f"TITLE: {a.get('title', '')}\n" + f"DESCRIPTION: {a.get('description', '')}\n" + f"OWNER: {a.get('owner_persona', '')} HORIZON: {a.get('horizon', '')} " + f"LEVER: {a.get('lever', '')} EFFORT: {a.get('effort', '')}\n" + f"EXPECTED_SAVINGS_FTE: {a.get('expected_savings_fte', '')} " + f"EXPECTED_SAVINGS_USD: {a.get('expected_savings_usd', '')}\n" + f"LINKED_TO_FINDING: {a.get('finding_id') in finding_ids}" + for a in actions + ] + answers = await _judge_all(ctx, SYSTEM, users, _Score) + scores = [a.score for a in answers if a.score is not None] + score = round(sum(scores) / len(scores), 4) if scores else None + return _scored(score, rated=len(scores)) + + +async def severity_calibration(item: dict, ctx: EvalContext) -> dict: + """Per-finding judgment of whether stated severity matches the evidence. + + Returns {score, miscalibrated, total, verdicts: {finding_id: under|over|calibrated}}; + score is the fraction of findings whose severity is calibrated (1 - miscalibrated / total). + """ + ev_idx = _evidence_index(item) + findings = item.get("findings", []) + users = [ + "Does the STATED SEVERITY match what the CITED EVIDENCE supports?\n" + 'Reply with ONLY {"calibration": "under" or "over" or "calibrated"}.\n\n' + f"STATED SEVERITY: {f.get('severity', '')} SCORE: {f.get('score', '')}\n" + f"FINDING: {f.get('description', '')}\n" + f"CITED EVIDENCE: {' || '.join(_cited_excerpts(f, ev_idx))}" + for f in findings + ] + answers = await _judge_all(ctx, SYSTEM, users, _Calibration) + verdicts: dict[str, str] = {} + miscalibrated = 0 + for f, a in zip(findings, answers, strict=False): + verdict = str(a.calibration).lower() + verdicts[f.get("id", "?")] = verdict + if verdict in ("under", "over"): + miscalibrated += 1 + return _scored( + round(1 - miscalibrated / len(findings), 4) if findings else None, + miscalibrated=miscalibrated, + total=len(findings), + verdicts=verdicts, + ) + + +async def answer_relevancy(item: dict, ctx: EvalContext) -> dict: + """RAGAS-style: does the output address the stated workspace intention? + + Returns {score} in [0,1] (score is None when the vote fails to coerce). + """ + user = ( + "Does the OUTPUT address the stated WORKSPACE INTENTION (on-topic, responsive)?\n" + 'Reply with ONLY {"score": }.\n\n' + f"WORKSPACE INTENTION: {_workspace_intention(item)}\n" + f"OUTPUT:\n{_output_text(item)}" + ) + answer = await ctx.client.judge(SYSTEM, user, _Score) + return _scored(answer.score) + + +async def surface_deduplication(item: dict, ctx: EvalContext) -> dict: + """Fraction of near-duplicate process-graph node pairs that are genuinely distinct. + + Returns {score, distinct, redundant, total, redundant_pairs}; score is the distinct rate + (distinct / total), None when there were no near-duplicate candidates to judge. + """ + pg = item.get("process_graph", {}) + procs = pg.get("processes", []) + + def _toks(node: dict) -> frozenset[str]: + return frozenset(node.get("name", "").lower().split()) + + per_surface_cap = 10 + candidates: list[tuple[str, dict, dict, str]] = [] + + if len(procs) >= 2: + pairs: list[tuple[float, dict, dict]] = [] + for i in range(len(procs)): + for j in range(i + 1, len(procs)): + a_t, b_t = _toks(procs[i]), _toks(procs[j]) + union = a_t | b_t + if not union: + continue + jac = len(a_t & b_t) / len(union) + if jac >= 0.30: + pairs.append((jac, procs[i], procs[j])) + pairs.sort(key=lambda x: x[0], reverse=True) + for _jac, a, b in pairs[:per_surface_cap]: + candidates.append(("process", a, b, "")) + + for surface_key, attr in (("activity", "activities"), ("decision", "decisions")): + all_pairs: list[tuple[float, dict, dict, str]] = [] + for proc in procs: + nodes = proc.get(attr, []) + proc_name = proc.get("name", "") + if len(nodes) < 2: + continue + for i in range(len(nodes)): + for j in range(i + 1, len(nodes)): + a_t, b_t = _toks(nodes[i]), _toks(nodes[j]) + union = a_t | b_t + if not union: + continue + jac = len(a_t & b_t) / len(union) + if jac >= 0.30: + all_pairs.append((jac, nodes[i], nodes[j], proc_name)) + all_pairs.sort(key=lambda x: x[0], reverse=True) + for _jac, a, b, proc_name in all_pairs[:per_surface_cap]: + candidates.append((surface_key, a, b, proc_name)) + + if not candidates: + return _scored(None, distinct=0, redundant=0, total=0, redundant_pairs=[]) + + users = [] + for surface, a, b, parent_proc in candidates: + ctx_line = f"\nPARENT PROCESS: {parent_proc}\n" if parent_proc else "" + users.append( + f"Are these two {surface} nodes genuinely DISTINCT process concepts, or is one a " + f"duplicate / sub-case / restatement of the other?\n" + f"{ctx_line}" + 'Reply with ONLY {"verdict": "DISTINCT" or "DUPLICATE", "reason": ""}.\n\n' + f"{surface.upper()} A: {a.get('name', '')} — {a.get('description', '')}\n" + f"{surface.upper()} B: {b.get('name', '')} — {b.get('description', '')}" + ) + + answers = await _judge_all(ctx, SYSTEM, users, _Verdict) + + distinct = 0 + redundant = 0 + redundant_pairs: list[dict] = [] + for (surface, a, b, _parent), answer in zip(candidates, answers, strict=False): + verdict = str(answer.verdict).upper() + if verdict == "DISTINCT": + distinct += 1 + else: + redundant += 1 + redundant_pairs.append( + { + "surface": surface, + "a": a.get("name", ""), + "b": b.get("name", ""), + "reason": str(answer.reason), + } + ) + + total = distinct + redundant + return _scored( + round(distinct / total, 4) if total else None, + distinct=distinct, + redundant=redundant, + total=total, + redundant_pairs=redundant_pairs, + ) + + +async def comparative_vs_champion(item: dict, ctx: EvalContext) -> dict | None: + """Pairwise MT-Bench-style review of candidate vs champion (advisory only). + + Returns None if item["champion"] is not present. + Returns {score, candidate, champion, more_consistent}; score is None (structured + pairwise comparison, no single aggregate). + """ + champion = item.get("champion") + if champion is None: + return None + user = ( + "Score the CANDIDATE and the CHAMPION outputs on five axes (1-5 each): " + "Coverage, Quality, Evidence, Actionability, Regression. Then say which is " + "more internally consistent.\n" + "Reply with ONLY " + '{"candidate": {"coverage": x, "quality": x, "evidence": x, "actionability": x, "regression": x}, ' + '"champion": {"coverage": x, "quality": x, "evidence": x, "actionability": x, "regression": x}, ' + '"more_consistent": "candidate" or "champion"}.\n\n' + f"CANDIDATE:\n{_output_text(item)}\n\n" + f"CHAMPION:\n{_output_text(champion)}" + ) + out = await ctx.client.judge(SYSTEM, user, _Comparison) + return _scored(None, candidate=out.candidate, champion=out.champion, more_consistent=out.more_consistent) + + +# ── flycanon custom metrics ─────────────────────────────────────────────────────── + + +async def _rag_score_once(item: dict, ctx: EvalContext) -> _RagScore | None: + """Single RAG scoring call returning a _RagScore (or None if item lacks Q/A).""" + question = item.get("question", "") + reference = item.get("reference", "") + answer = item.get("answer", "") + if not question or not answer: + return None + user = f"QUESTION: {question}\nREFERENCE: {reference}\nANSWER: {answer}\n\n{RUBRIC}" + return await ctx.client.judge(SYSTEM_RAG, user, _RagScore) + + +async def contains_answer(item: dict, ctx: EvalContext) -> dict | None: + """Flycanon: does the answer contain the correct information from the reference? + + Returns {score} in [0,1], or None if the item lacks question/answer. + """ + result = await _rag_score_once(item, ctx) + if result is None or result.contains_answer is None: + return None + return _scored(round(result.contains_answer, 4)) + + +async def addresses_question(item: dict, ctx: EvalContext) -> dict | None: + """Flycanon: does the answer directly address what the question is asking? + + Returns {score} in [0,1], or None if the item lacks question/answer. + """ + result = await _rag_score_once(item, ctx) + if result is None or result.addresses_question is None: + return None + return _scored(round(result.addresses_question, 4)) + + +# ── RAGAS metrics ───────────────────────────────────────────────────────────────── +# ragas/langchain imports are inline inside _sync() since ragas is optional. + + +def _make_ragas_sample(item: dict): + """Build a RAGAS SingleTurnSample from an item dict (ragas import inline).""" + from ragas import SingleTurnSample # type: ignore[import] # noqa: PLC0415 + + return SingleTurnSample( + user_input=item.get("question", ""), + response=item.get("answer", ""), + reference=item.get("reference", ""), + retrieved_contexts=item.get("contexts", []), + ) + + +def _make_ragas_llm(ctx: EvalContext): + """Build a LangChain LLM wrapper for RAGAS (langchain import inline).""" + provider, model = ctx.client.provider, ctx.client.model + if provider == "anthropic": + from langchain_anthropic import ChatAnthropic # type: ignore[import] # noqa: PLC0415 + + api_key = os.environ.get("ANTHROPIC_API_KEY", "") + return ChatAnthropic(model=model, api_key=api_key, temperature=0.0) # type: ignore[call-arg,arg-type] + if provider == "openai": + from langchain_openai import ChatOpenAI # type: ignore[import] # noqa: PLC0415 + + api_key = os.environ.get("OPENAI_API_KEY", "") + return ChatOpenAI(model=model, api_key=api_key, temperature=0.0) # type: ignore[call-arg,arg-type] + if provider == "azure": + from langchain_openai import AzureChatOpenAI # type: ignore[import] # noqa: PLC0415 + + return AzureChatOpenAI( # type: ignore[call-arg] + azure_deployment=model, + azure_endpoint=os.environ.get("AZURE_OPENAI_ENDPOINT", ""), + api_key=os.environ.get("AZURE_OPENAI_API_KEY", ""), # type: ignore[arg-type] + api_version=os.environ.get("AZURE_OPENAI_API_VERSION", "2024-02-01"), + temperature=0.0, + ) + if provider == "ollama": + from langchain_ollama import ChatOllama # type: ignore[import] # noqa: PLC0415 + + return ChatOllama(model=model, temperature=0.0) + raise ValueError(f"RAGAS: unsupported provider {provider!r}") + + +def _build_embeddings(ctx: EvalContext): + """Wrap the framework embedder (``ctx.embedder``) for RAGAS. + + RAGAS consumes a LangChain ``Embeddings`` via ``LangchainEmbeddingsWrapper``; + we feed it a thin adapter over the fireflyframework_agentic ``BaseEmbedder`` so + RAGAS uses the same embedder (and provider) as the rest of the pipeline. Build + one with :func:`fireflyframework_agentic.evaluation.build_embedder`. + """ + from langchain_core.embeddings import Embeddings # type: ignore[import] # noqa: PLC0415 + from ragas.embeddings import LangchainEmbeddingsWrapper # type: ignore[import] # noqa: PLC0415 + + embedder = ctx.embedder + if embedder is None: + raise ValueError( + "RAGAS metrics need an embedder; set EvalContext.embedder=build_embedder(':')" + ) + + class _FrameworkEmbeddings(Embeddings): + async def aembed_documents(self, texts: list[str]) -> list[list[float]]: + return (await embedder.embed(texts)).embeddings + + async def aembed_query(self, text: str) -> list[float]: + return await embedder.embed_one(text) + + def embed_documents(self, texts: list[str]) -> list[list[float]]: + return asyncio.run(self.aembed_documents(texts)) + + def embed_query(self, text: str) -> list[float]: + return asyncio.run(self.aembed_query(text)) + + return LangchainEmbeddingsWrapper(_FrameworkEmbeddings()) + + +async def _ragas_score(metric_name: str, item: dict, ctx: EvalContext) -> float | None: + """Run a single named RAGAS metric and return its float score (or None).""" + + def _sync(): + from ragas import evaluate # type: ignore[import] # noqa: PLC0415 + from ragas.dataset_schema import EvaluationDataset # type: ignore[import] # noqa: PLC0415 + from ragas.metrics import ( # type: ignore[import] # noqa: PLC0415 + AnswerCorrectness, + AnswerRelevancy, + ContextPrecision, + ContextRecall, + Faithfulness, + ) + + _metrics_map = { + "answer_correctness": AnswerCorrectness, + "answer_relevancy_ragas": AnswerRelevancy, + "ragas_faithfulness": Faithfulness, + "context_recall": ContextRecall, + "context_precision": ContextPrecision, + } + metric_cls = _metrics_map.get(metric_name) + if metric_cls is None: + return None + + llm = _make_ragas_llm(ctx) + embeddings = _build_embeddings(ctx) + metric = metric_cls(llm=llm, embeddings=embeddings) + sample = _make_ragas_sample(item) + dataset = EvaluationDataset(samples=[sample]) + result = evaluate(dataset=dataset, metrics=[metric]) + df = result.to_pandas() # type: ignore[attr-defined] + col = df.columns[df.columns.str.contains(metric_name.replace("ragas_", "").replace("_ragas", ""), case=False)] + if col.empty: + return None + val = df[col[0]].iloc[0] + if val is None or (isinstance(val, float) and math.isnan(val)): + return None + return round(float(val), 4) + + loop = asyncio.get_event_loop() + return await loop.run_in_executor(None, _sync) + + +async def answer_correctness(item: dict, ctx: EvalContext) -> dict | None: + """RAGAS answer correctness (semantic F1 against reference). Returns {score} or None.""" + val = await _ragas_score("answer_correctness", item, ctx) + return _scored(val) if val is not None else None + + +async def ragas_faithfulness(item: dict, ctx: EvalContext) -> dict | None: + """RAGAS faithfulness: fraction of the answer's atomic claims grounded in contexts. + + Runs the ragas library's Faithfulness metric on a RAG Q&A item. RAGAS first decomposes + ``answer`` into atomic claims (one LLM pass), then verifies each claim against the + retrieved ``contexts`` (a verdict per claim); the score is supported_claims / + total_claims. Returns {score} in [0, 1], or None when it cannot be computed. + + Differs from the custom ``faithfulness``: that one judges each discovery FINDING as a + whole against its own CITED excerpts (not retrieved contexts) and reports a + {score, supported, total, unsupported_ids} tally over findings. This one grades a + free-text RAG answer's grounding in its contexts. + """ + val = await _ragas_score("ragas_faithfulness", item, ctx) + return _scored(val) if val is not None else None + + +async def context_recall(item: dict, ctx: EvalContext) -> dict | None: + """RAGAS context recall (reference coverage by retrieved contexts). Returns {score} or None.""" + val = await _ragas_score("context_recall", item, ctx) + return _scored(val) if val is not None else None + + +async def context_precision(item: dict, ctx: EvalContext) -> dict | None: + """RAGAS context precision (retrieved contexts relevant to the question). Returns {score} or None.""" + val = await _ragas_score("context_precision", item, ctx) + return _scored(val) if val is not None else None + + +# ── metric families ────────────────────────────────────────────────────────────── + +# Domain-agnostic LLM / RAG answer-quality metrics. +BASIC_METRICS: tuple[str, ...] = ( + "contains_answer", + "addresses_question", + "answer_correctness", + "ragas_faithfulness", + "context_recall", + "context_precision", +) + +# Flyradar process-mining discovery-report metrics. +PROCESS_MINING_METRICS: tuple[str, ...] = ( + "source_coverage", + "excerpt_fill_rate", + "semantic_recovery", + "faithfulness", + "numeric_temporal_fidelity", + "citation_relevance", + "nc_semantic_precision", + "fabricated_entity", + "contradiction", + "open_gap", + "actionability", + "severity_calibration", + "answer_relevancy", + "surface_deduplication", + "comparative_vs_champion", +) + +_METRIC_FNS: dict[str, Metric] = { + "source_coverage": source_coverage, + "excerpt_fill_rate": excerpt_fill_rate, + "semantic_recovery": semantic_recovery, + "faithfulness": faithfulness, + "numeric_temporal_fidelity": numeric_temporal_fidelity, + "citation_relevance": citation_relevance, + "nc_semantic_precision": nc_semantic_precision, + "fabricated_entity": fabricated_entity, + "contradiction": contradiction, + "open_gap": open_gap, + "actionability": actionability, + "severity_calibration": severity_calibration, + "answer_relevancy": answer_relevancy, + "surface_deduplication": surface_deduplication, + "comparative_vs_champion": comparative_vs_champion, + "contains_answer": contains_answer, + "addresses_question": addresses_question, + "answer_correctness": answer_correctness, + "ragas_faithfulness": ragas_faithfulness, + "context_recall": context_recall, + "context_precision": context_precision, +} + + +def _selected_metric_names(metrics: str) -> tuple[str, ...]: + if metrics == "basic": + return BASIC_METRICS + if metrics == "process_mining": + return PROCESS_MINING_METRICS + if metrics == "all": + return BASIC_METRICS + PROCESS_MINING_METRICS + raise ValueError(f"metrics must be 'all', 'basic', or 'process_mining'; got {metrics!r}") + + +# ── orchestrator ───────────────────────────────────────────────────────────────── + + +async def run_judge( + item: dict, + ctx: EvalContext, + *, + metrics: str = "all", + pipeline_model: str = "", +) -> AdvisoryReport: + """Run the selected metric family concurrently and return an AdvisoryReport. + + ``metrics`` selects which family runs: ``"basic"`` (domain-agnostic LLM/RAG + answer-quality), ``"process_mining"`` (flyradar discovery-report), or ``"all"``. + Best-effort: never raises. Failing metrics append to report.errors. + """ + report = AdvisoryReport( + judge_model=ctx.client.model_spec, + same_provider_caveat=same_provider(pipeline_model, ctx.client.model_spec), + ) + selected = [(name, _METRIC_FNS[name]) for name in _selected_metric_names(metrics)] + + async def _run_one(name: str, fn: Metric) -> None: + try: + result = await fn(item, ctx) + if result is not None: + report.metrics[name] = result + except Exception as exc: + report.errors.append(f"{name}: {type(exc).__name__}: {exc}") + + await asyncio.gather(*[_run_one(name, fn) for name, fn in selected]) + return report diff --git a/fireflyframework_agentic/evaluation/retrieval_metrics.py b/fireflyframework_agentic/evaluation/retrieval_metrics.py new file mode 100644 index 00000000..b85a4ba3 --- /dev/null +++ b/fireflyframework_agentic/evaluation/retrieval_metrics.py @@ -0,0 +1,174 @@ +# Copyright 2026 Firefly Software Foundation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Deterministic IR evaluation metrics for ranked retrieval results (no LLM, no network). + +Each metric is a plain function that takes a list of result rows and returns a +float — the same design as scikit-learn or MS MARCO evaluation scripts. + +Result row schema (dict):: + + { + "retrieved": [{"rank": int, "source_id": str, "is_gold": bool}, ...], + "gold": [str, ...], # gold source identifiers + # optional: + "no_answer": bool, # model refused / produced no answer + "answer": str, # used for no_answer detection when no_answer absent + "citations": [{"is_gold": bool}, ...], + } + +Individual metrics:: + + hit_at_k(results, k) -> float + recall_at_k(results, k) -> float + precision_at_k(results, k) -> float + mrr(results, k=10) -> float + map_score(results, k=10) -> float + ndcg(results, k=10) -> float + no_answer_rate(results) -> float | None + citation_precision(results) -> float | None +""" + +from __future__ import annotations + +import math + + +def _dedup(retrieved: list[dict]) -> list[dict]: + """Return one entry per source (best-ranked chunk wins), ranks re-densified 1..N. + + De-duplicating by source can leave gaps in the raw chunk ranks — e.g. four + chunks of one source occupy ranks 1-4 and the next source sits at rank 5. Without + re-numbering, the rank metrics would treat that source as document-rank 5 instead + of 2 and understate recall/precision/nDCG/MRR/MAP. Renumber the survivors to + contiguous document positions so the metrics are document-level. + """ + seen: set[str] = set() + out: list[dict] = [] + for r in sorted(retrieved, key=lambda x: x["rank"]): + key = r.get("source_id") or "|".join(r.get("identities", [])) + if key not in seen: + seen.add(key) + out.append(r) + return [{**r, "rank": i} for i, r in enumerate(out, start=1)] + + +def _ndcg_single(retrieved: list[dict], n_gold: int, k: int = 10) -> float: + dcg = sum(1.0 / math.log2(r["rank"] + 1) for r in retrieved if r.get("is_gold") and r["rank"] <= k) + ideal = sum(1.0 / math.log2(i + 2) for i in range(min(n_gold, k))) + return dcg / ideal if ideal else 0.0 + + +def _ap_single(retrieved: list[dict], n_gold: int, k: int = 10) -> float: + hits, precisions = 0, [] + for r in sorted(retrieved, key=lambda x: x["rank"]): + if r["rank"] > k: + break + if r.get("is_gold"): + hits += 1 + precisions.append(hits / r["rank"]) + return sum(precisions) / n_gold if n_gold else 0.0 + + +def hit_at_k(results: list[dict], k: int) -> float: + """Fraction of queries where at least one gold document appears in top-k.""" + if not results: + return 0.0 + hits = 0 + for row in results: + retrieved = _dedup(row["retrieved"]) + gold_ranks = [r["rank"] for r in retrieved if r.get("is_gold")] + if any(g <= k for g in gold_ranks): + hits += 1 + return round(hits / len(results), 4) + + +def recall_at_k(results: list[dict], k: int) -> float: + """Mean fraction of gold documents found in top-k.""" + if not results: + return 0.0 + total = 0.0 + for row in results: + retrieved = _dedup(row["retrieved"]) + n_gold = max(len(set(row["gold"])), 1) + gold_ranks = [r["rank"] for r in retrieved if r.get("is_gold")] + total += len([g for g in gold_ranks if g <= k]) / n_gold + return round(total / len(results), 4) + + +def precision_at_k(results: list[dict], k: int) -> float: + """Mean fraction of top-k results that are gold.""" + if not results: + return 0.0 + total = 0.0 + for row in results: + retrieved = _dedup(row["retrieved"]) + gold_ranks = [r["rank"] for r in retrieved if r.get("is_gold")] + total += len([g for g in gold_ranks if g <= k]) / k + return round(total / len(results), 4) + + +def mrr(results: list[dict], k: int = 10) -> float: + """Mean reciprocal rank of the first gold hit (up to k).""" + if not results: + return 0.0 + total = 0.0 + for row in results: + retrieved = _dedup(row["retrieved"]) + gold_ranks = sorted(r["rank"] for r in retrieved if r.get("is_gold") and r["rank"] <= k) + total += 1.0 / gold_ranks[0] if gold_ranks else 0.0 + return round(total / len(results), 4) + + +def map_score(results: list[dict], k: int = 10) -> float: + """Mean average precision at k.""" + if not results: + return 0.0 + total = 0.0 + for row in results: + retrieved = _dedup(row["retrieved"]) + n_gold = max(len(set(row["gold"])), 1) + total += _ap_single(retrieved, n_gold, k) + return round(total / len(results), 4) + + +def ndcg(results: list[dict], k: int = 10) -> float: + """Mean normalised discounted cumulative gain at k.""" + if not results: + return 0.0 + total = 0.0 + for row in results: + retrieved = _dedup(row["retrieved"]) + n_gold = max(len(set(row["gold"])), 1) + total += _ndcg_single(retrieved, n_gold, k) + return round(total / len(results), 4) + + +def no_answer_rate(results: list[dict]) -> float | None: + """Fraction of queries where the model produced no answer. None if no results.""" + if not results: + return None + count = sum(1 for row in results if row.get("no_answer") or not (row.get("answer") or "").strip()) + return round(count / len(results), 4) + + +def citation_precision(results: list[dict]) -> float | None: + """Precision of in-answer citations vs gold set. None if no citations present.""" + num = den = 0.0 + for row in results: + cites = row.get("citations", []) + if cites: + num += sum(1 for c in cites if c.get("is_gold")) + den += len(cites) + return round(num / den, 4) if den else None diff --git a/pyproject.toml b/pyproject.toml index 92b31055..ecb0e767 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "fireflyframework-agentic" -version = "26.06.13" +version = "26.06.14" description = "A GenAI metaframework built on Pydantic AI for building production-grade GenAI applications with agents, reasoning patterns, prompt engineering, observability, and more." readme = "README.md" license = { text = "Apache-2.0" } @@ -134,6 +134,13 @@ binary = [ all = [ "fireflyframework-agentic[postgres,mongodb,security,embeddings,openai-embeddings,cohere-embeddings,google-embeddings,mistral-embeddings,voyage-embeddings,azure-embeddings,bedrock-embeddings,ollama-embeddings,vectorstores-chroma,vectorstores-pinecone,vectorstores-qdrant,vectorstores-pgvector,vectorstores-sqlite-vec,watch,binary,script-execution]", ] +evaluation = [ + "numpy>=1.26.0", + "ragas>=0.2", + "langchain-anthropic>=0.3", + "langchain-ollama>=0.3", + "langchain-openai>=0.3", +] dev = [ "pytest>=8.3.0", "pytest-asyncio>=0.24.0", diff --git a/tests/unit/evaluation/__init__.py b/tests/unit/evaluation/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/tests/unit/evaluation/test_embedder.py b/tests/unit/evaluation/test_embedder.py new file mode 100644 index 00000000..803d2cdf --- /dev/null +++ b/tests/unit/evaluation/test_embedder.py @@ -0,0 +1,44 @@ +# Copyright 2026 Firefly Software Foundation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Unit tests for evaluation.embedder.build_embedder.""" + +from __future__ import annotations + +import pytest + +from fireflyframework_agentic.embeddings.providers.ollama import OllamaEmbedder +from fireflyframework_agentic.evaluation import build_embedder + + +def test_build_embedder_ollama_returns_framework_embedder(): + embedder = build_embedder("ollama:nomic-embed-text") + assert isinstance(embedder, OllamaEmbedder) + assert embedder.model == "nomic-embed-text" + + +def test_build_embedder_honours_ollama_host(monkeypatch): + monkeypatch.setenv("OLLAMA_HOST", "http://example:1234") + embedder = build_embedder("ollama:nomic-embed-text") + assert embedder._base_url == "http://example:1234" + + +def test_build_embedder_requires_provider_prefix(): + with pytest.raises(ValueError, match=":"): + build_embedder("nomic-embed-text") + + +def test_build_embedder_rejects_unknown_provider(): + with pytest.raises(ValueError, match="unknown embedding provider"): + build_embedder("bogus:model") diff --git a/tests/unit/evaluation/test_judge.py b/tests/unit/evaluation/test_judge.py new file mode 100644 index 00000000..7083a6b0 --- /dev/null +++ b/tests/unit/evaluation/test_judge.py @@ -0,0 +1,330 @@ +from unittest.mock import MagicMock + +import pytest + +from fireflyframework_agentic.agents import FireflyAgent +from fireflyframework_agentic.evaluation.judge import ( + EvalContext, + JudgeClient, + _RagScore, + _Verdict, + addresses_question, + contains_answer, + excerpt_fill_rate, + faithfulness, + run_judge, + source_coverage, +) + + +def make_ctx(responses: list[dict]) -> EvalContext: + client = MagicMock(spec=JudgeClient) + client.model_spec = "anthropic:claude-sonnet-4-6" + client.provider = "anthropic" + client.model = "claude-sonnet-4-6" + call_iter = iter(responses) + + async def mock_judge(system, user, output_type, max_tokens=1024): + return output_type(**next(call_iter)) + + client.judge = mock_judge + return EvalContext(client=client) + + +# ── contains_answer ────────────────────────────────────────────────────────────── + + +@pytest.mark.asyncio +async def test_contains_answer_present(): + ctx = make_ctx([{"contains_answer": 1.0, "addresses_question": 1.0}]) + item = {"question": "Q", "reference": "R", "answer": "A"} + score = await contains_answer(item, ctx) + assert score["score"] == 1.0 + + +@pytest.mark.asyncio +async def test_contains_answer_absent(): + ctx = make_ctx([{"contains_answer": 0.0, "addresses_question": 0.5}]) + item = {"question": "Q", "reference": "R", "answer": "wrong"} + score = await contains_answer(item, ctx) + assert score["score"] == 0.0 + + +@pytest.mark.asyncio +async def test_contains_answer_partial(): + ctx = make_ctx([{"contains_answer": 0.5, "addresses_question": 0.8}]) + item = {"question": "Q", "reference": "R", "answer": "partial"} + score = await contains_answer(item, ctx) + assert score["score"] == 0.5 + + +@pytest.mark.asyncio +async def test_contains_answer_missing_question_returns_none(): + ctx = make_ctx([]) + item = {"reference": "R", "answer": "A"} + score = await contains_answer(item, ctx) + assert score is None + + +# ── addresses_question ─────────────────────────────────────────────────────────── + + +@pytest.mark.asyncio +async def test_addresses_question_yes(): + ctx = make_ctx([{"contains_answer": 0.5, "addresses_question": 1.0}]) + item = {"question": "Q", "reference": "R", "answer": "A"} + score = await addresses_question(item, ctx) + assert score["score"] == 1.0 + + +@pytest.mark.asyncio +async def test_addresses_question_no(): + ctx = make_ctx([{"contains_answer": 0.0, "addresses_question": 0.0}]) + item = {"question": "Q", "reference": "R", "answer": "irrelevant"} + score = await addresses_question(item, ctx) + assert score["score"] == 0.0 + + +@pytest.mark.asyncio +async def test_addresses_question_missing_answer_returns_none(): + ctx = make_ctx([]) + item = {"question": "Q", "reference": "R"} + score = await addresses_question(item, ctx) + assert score is None + + +# ── faithfulness ───────────────────────────────────────────────────────────────── + + +@pytest.mark.asyncio +async def test_faithfulness_all_supported(): + # One finding with cited evidence, judge says SUPPORTED. + ctx = make_ctx([{"verdict": "SUPPORTED", "reason": "matches"}]) + item = { + "findings": [ + { + "id": "F1", + "description": "The process takes 3 days.", + "evidence_refs": [{"evidence_id": "E1"}], + } + ], + "evidence_index": [{"id": "E1", "locator": "doc.pdf#1", "excerpt": "The process takes 3 days as documented."}], + } + result = await faithfulness(item, ctx) + assert result["supported"] == 1 + assert result["total"] == 1 + assert result["unsupported_ids"] == [] + + +@pytest.mark.asyncio +async def test_faithfulness_not_supported(): + ctx = make_ctx([{"verdict": "NOT_SUPPORTED", "reason": "contradicts"}]) + item = { + "findings": [ + { + "id": "F1", + "description": "The process takes 45 days.", + "evidence_refs": [{"evidence_id": "E1"}], + } + ], + "evidence_index": [{"id": "E1", "locator": "doc.pdf#1", "excerpt": "The process takes 3 days."}], + } + result = await faithfulness(item, ctx) + assert result["supported"] == 0 + assert result["total"] == 1 + assert "F1" in result["unsupported_ids"] + + +@pytest.mark.asyncio +async def test_faithfulness_no_cited_evidence(): + # Finding with no evidence_refs -> counted as unsupported without LLM call. + ctx = make_ctx([]) + item = { + "findings": [{"id": "F1", "description": "Something.", "evidence_refs": []}], + "evidence_index": [], + } + result = await faithfulness(item, ctx) + assert result["supported"] == 0 + assert result["total"] == 1 + assert "F1" in result["unsupported_ids"] + + +# ── source_coverage ─────────────────────────────────────────────────────────────── + + +@pytest.mark.asyncio +async def test_source_coverage_all_cited(): + ctx = make_ctx([]) + item = { + "findings": [ + { + "id": "F1", + "description": "X", + "evidence_refs": [{"evidence_id": "E1"}], + } + ], + "evidence_index": [{"id": "E1", "locator": "doc.pdf#section1", "excerpt": "text"}], + } + result = await source_coverage(item, ctx) + assert result["cited"] == 1 + assert result["total"] == 1 + assert result["orphaned"] == [] + + +@pytest.mark.asyncio +async def test_source_coverage_orphaned(): + ctx = make_ctx([]) + item = { + "findings": [{"id": "F1", "description": "X", "evidence_refs": []}], + "evidence_index": [ + {"id": "E1", "locator": "doc1.pdf#p1", "excerpt": "text"}, + {"id": "E2", "locator": "doc2.pdf#p2", "excerpt": "text2"}, + ], + } + result = await source_coverage(item, ctx) + assert result["cited"] == 0 + assert result["total"] == 2 + assert len(result["orphaned"]) == 2 + + +@pytest.mark.asyncio +async def test_source_coverage_stem_dedup(): + # Two evidence items from the same file (different fragments) -> 1 source stem. + ctx = make_ctx([]) + item = { + "findings": [ + { + "id": "F1", + "description": "X", + "evidence_refs": [{"evidence_id": "E1"}], + } + ], + "evidence_index": [ + {"id": "E1", "locator": "doc.pdf#section1", "excerpt": "text1"}, + {"id": "E2", "locator": "doc.pdf#section2", "excerpt": "text2"}, + ], + } + result = await source_coverage(item, ctx) + # Both E1 and E2 share "doc.pdf" stem -> 1 total stem. + assert result["total"] == 1 + # E1 is cited -> that stem is covered. + assert result["cited"] == 1 + + +# ── excerpt_fill_rate ────────────────────────────────────────────────────────────── + + +@pytest.mark.asyncio +async def test_excerpt_fill_rate_full(): + ctx = make_ctx([]) + item = { + "evidence_index": [ + {"id": "E1", "excerpt": "has content"}, + {"id": "E2", "excerpt": "also has content"}, + ] + } + result = await excerpt_fill_rate(item, ctx) + assert result["populated"] == 2 + assert result["total"] == 2 + + +@pytest.mark.asyncio +async def test_excerpt_fill_rate_partial(): + ctx = make_ctx([]) + item = { + "evidence_index": [ + {"id": "E1", "excerpt": "has content"}, + {"id": "E2", "excerpt": ""}, + {"id": "E3", "excerpt": " "}, + ] + } + result = await excerpt_fill_rate(item, ctx) + assert result["populated"] == 1 + assert result["total"] == 3 + + +@pytest.mark.asyncio +async def test_excerpt_fill_rate_empty(): + ctx = make_ctx([]) + item = {"evidence_index": []} + result = await excerpt_fill_rate(item, ctx) + assert result["populated"] == 0 + assert result["total"] == 0 + + +# ── JudgeClient (FireflyAgent-backed) ───────────────────────────────────────────── + + +def test_judge_client_builds_and_caches_agent(monkeypatch): + monkeypatch.setenv("ANTHROPIC_API_KEY", "dummy") + client = JudgeClient("anthropic:claude-sonnet-4-6") + a1 = client._agent("sys", _Verdict, 1024) + a2 = client._agent("sys", _Verdict, 1024) + assert isinstance(a1, FireflyAgent) + assert a1 is a2 # cached per (system, output_type, max_tokens) + + +@pytest.mark.asyncio +async def test_faithfulness_propagates_judge_failure(): + # A failed judge call must NOT be silently scored as a verdict — it propagates. + client = MagicMock(spec=JudgeClient) + + async def boom(system, user, output_type, max_tokens=1024): + raise RuntimeError("API down") + + client.judge = boom + ctx = EvalContext(client=client, runs=1) + item = { + "findings": [{"id": "F1", "description": "x", "evidence_refs": [{"evidence_id": "E1"}]}], + "evidence_index": [{"id": "E1", "excerpt": "y"}], + } + with pytest.raises(RuntimeError): + await faithfulness(item, ctx) + + +@pytest.mark.asyncio +async def test_run_judge_aggregates_and_captures_errors(): + client = MagicMock(spec=JudgeClient) + client.model_spec = "anthropic:claude-sonnet-4-6" + + async def mock_judge(system, user, output_type, max_tokens=1024): + return output_type() + + client.judge = mock_judge + ctx = EvalContext(client=client) + report = await run_judge( + {"question": "Q", "reference": "R", "answer": "A"}, ctx, pipeline_model="anthropic:claude-sonnet-4-6" + ) + assert report.judge_model == "anthropic:claude-sonnet-4-6" + assert report.same_provider_caveat is True + assert "source_coverage" in report.metrics # process-mining metric runs under "all" + assert isinstance(report.errors, list) # best-effort: never raises + + +@pytest.mark.asyncio +async def test_run_judge_metric_family_selection(): + client = MagicMock(spec=JudgeClient) + client.model_spec = "anthropic:claude-sonnet-4-6" + + async def mock_judge(system, user, output_type, max_tokens=1024): + return output_type() + + async def mock_judge_rag(system, user, output_type, max_tokens=1024): + if output_type is _RagScore: + return _RagScore(contains_answer=1.0, addresses_question=1.0) + return output_type() + + client.judge = mock_judge_rag + ctx = EvalContext(client=client) + item = {"question": "Q", "reference": "R", "answer": "A"} + + basic = await run_judge(item, ctx, metrics="basic") + assert "contains_answer" in basic.metrics + assert "source_coverage" not in basic.metrics # process-mining excluded + + pm = await run_judge(item, ctx, metrics="process_mining") + assert "source_coverage" in pm.metrics + assert "contains_answer" not in pm.metrics # basic excluded + + with pytest.raises(ValueError, match="metrics must be"): + await run_judge(item, ctx, metrics="bogus") diff --git a/tests/unit/evaluation/test_retrieval_metrics.py b/tests/unit/evaluation/test_retrieval_metrics.py new file mode 100644 index 00000000..0983abdd --- /dev/null +++ b/tests/unit/evaluation/test_retrieval_metrics.py @@ -0,0 +1,194 @@ +# Copyright 2026 Firefly Software Foundation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Unit tests for evaluation.retrieval_metrics.""" + +from __future__ import annotations + +from fireflyframework_agentic.evaluation.retrieval_metrics import ( + citation_precision, + hit_at_k, + map_score, + mrr, + ndcg, + no_answer_rate, + precision_at_k, + recall_at_k, +) + + +def _row(gold_rank: int | None = None, total: int = 5, n_gold: int = 1) -> dict: + retrieved = [] + for rank in range(1, total + 1): + retrieved.append({"rank": rank, "source_id": f"doc-{rank}", "is_gold": rank == gold_rank}) + gold_ids = [f"doc-{gold_rank}"] if gold_rank is not None else [] + return {"retrieved": retrieved, "gold": gold_ids * n_gold} + + +# ── hit_at_k ────────────────────────────────────────────────────────────────── + + +def test_hit_at_k_gold_at_rank1(): + assert hit_at_k([_row(gold_rank=1)], k=1) == 1.0 + + +def test_hit_at_k_miss_at_rank1(): + assert hit_at_k([_row(gold_rank=2)], k=1) == 0.0 + + +def test_hit_at_k_gold_at_rank5(): + assert hit_at_k([_row(gold_rank=5)], k=5) == 1.0 + + +def test_hit_at_k_gold_at_rank10(): + assert hit_at_k([_row(gold_rank=10, total=10)], k=10) == 1.0 + + +def test_hit_at_k_empty(): + assert hit_at_k([], k=5) == 0.0 + + +# ── recall_at_k ─────────────────────────────────────────────────────────────── + + +def test_recall_at_k_full_when_gold_at_rank1(): + assert recall_at_k([_row(gold_rank=1, n_gold=1)], k=1) == 1.0 + + +def test_recall_at_k_zero_when_gold_outside_k(): + assert recall_at_k([_row(gold_rank=5)], k=1) == 0.0 + + +def test_recall_at_k_increases_with_k(): + rows = [_row(gold_rank=3)] + assert recall_at_k(rows, k=1) <= recall_at_k(rows, k=5) <= recall_at_k(rows, k=10) + + +# ── precision_at_k ──────────────────────────────────────────────────────────── + + +def test_precision_at_k_gold_at_rank1(): + assert precision_at_k([_row(gold_rank=1)], k=1) == 1.0 + + +def test_precision_at_k_decreases_when_k_larger(): + rows = [_row(gold_rank=1)] + assert precision_at_k(rows, k=5) < precision_at_k(rows, k=1) + + +# ── mrr ─────────────────────────────────────────────────────────────────────── + + +def test_mrr_gold_at_rank1(): + assert mrr([_row(gold_rank=1)]) == 1.0 + + +def test_mrr_gold_at_rank2(): + assert abs(mrr([_row(gold_rank=2)]) - 0.5) < 1e-9 + + +def test_mrr_no_gold(): + assert mrr([_row(gold_rank=None)]) == 0.0 + + +def test_mrr_average_across_queries(): + rows = [_row(gold_rank=1), _row(gold_rank=2)] + assert abs(mrr(rows) - 0.75) < 1e-3 + + +# ── ndcg ────────────────────────────────────────────────────────────────────── + + +def test_ndcg_gold_at_rank1(): + assert abs(ndcg([_row(gold_rank=1, n_gold=1)]) - 1.0) < 1e-9 + + +def test_ndcg_less_than_1_when_not_at_rank1(): + score = ndcg([_row(gold_rank=3, n_gold=1)]) + assert 0.0 < score < 1.0 + + +def test_ndcg_zero_when_no_gold(): + assert ndcg([_row(gold_rank=None)]) == 0.0 + + +# ── map_score ───────────────────────────────────────────────────────────────── + + +def test_map_score_perfect_when_gold_at_rank1(): + assert map_score([_row(gold_rank=1, n_gold=1)]) == 1.0 + + +def test_map_score_zero_when_no_gold(): + assert map_score([_row(gold_rank=None)]) == 0.0 + + +# ── no_answer_rate ──────────────────────────────────────────────────────────── + + +def test_no_answer_rate_zero_when_answer_present(): + rows = [{**_row(gold_rank=1), "answer": "some answer"}] + assert no_answer_rate(rows) == 0.0 + + +def test_no_answer_rate_one_when_no_answer_field(): + assert no_answer_rate([_row(gold_rank=1)]) == 1.0 + + +def test_no_answer_rate_none_when_empty(): + assert no_answer_rate([]) is None + + +# ── citation_precision ──────────────────────────────────────────────────────── + + +def test_citation_precision_none_when_no_citations(): + assert citation_precision([_row(gold_rank=1)]) is None + + +def test_citation_precision_1_when_all_gold(): + rows = [{**_row(gold_rank=1), "citations": [{"is_gold": True}, {"is_gold": True}]}] + assert citation_precision(rows) == 1.0 + + +def test_citation_precision_half_when_half_gold(): + rows = [{**_row(gold_rank=1), "citations": [{"is_gold": True}, {"is_gold": False}]}] + assert citation_precision(rows) == 0.5 + + +# ── _dedup rank re-densification ─────────────────────────────────────────────── + + +def _multichunk_row() -> dict: + # 4 chunks of one (non-gold) source at ranks 1-4, gold source at chunk-rank 5. + retrieved = [{"rank": r, "source_id": "doc-A", "is_gold": False} for r in range(1, 5)] + retrieved.append({"rank": 5, "source_id": "doc-B", "is_gold": True}) + return {"retrieved": retrieved, "gold": ["doc-B"]} + + +def test_multichunk_source_redensifies_to_document_rank(): + rows = [_multichunk_row()] + # After dedup, doc-B is the 2nd distinct document (rank 2), not rank 5. + assert recall_at_k(rows, k=3) == 1.0 # was 0.0 before the re-densify fix + assert precision_at_k(rows, k=2) == 0.5 # was 0.0 + assert ndcg(rows, k=3) == 0.6309 # gold at doc-rank 2: 1/log2(3) + assert mrr(rows) == 0.5 # was 0.2 + + +# ── no_answer_rate with None answer ──────────────────────────────────────────── + + +def test_no_answer_rate_handles_none_answer(): + rows = [{**_row(gold_rank=1), "answer": None}, {**_row(gold_rank=1), "answer": "real"}] + assert no_answer_rate(rows) == 0.5 diff --git a/uv.lock b/uv.lock index dc3cf737..b519af94 100644 --- a/uv.lock +++ b/uv.lock @@ -194,6 +194,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/da/42/e921fccf5015463e32a3cf6ee7f980a6ed0f395ceeaa45060b61d86486c2/anyio-4.13.0-py3-none-any.whl", hash = "sha256:08b310f9e24a9594186fd75b4f73f4a4152069e3853f1ed8bfbf58369f4ad708", size = 114353, upload-time = "2026-03-24T12:59:08.246Z" }, ] +[[package]] +name = "appdirs" +version = "1.4.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d7/d8/05696357e0311f5b5c316d7b95f46c669dd9c15aaeecbb48c7d0aeb88c40/appdirs-1.4.4.tar.gz", hash = "sha256:7d5d0167b2b1ba821647616af46a749d1c653740dd0d2415100fe26e27afdf41", size = 13470, upload-time = "2020-05-11T07:59:51.037Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3b/00/2344469e2084fb287c2e0b57b72910309874c3245463acd6cf5e3db69324/appdirs-1.4.4-py2.py3-none-any.whl", hash = "sha256:a841dacd6b99318a741b166adb07e19ee71a274450e68237b4650ca1055ab128", size = 9566, upload-time = "2020-05-11T07:59:49.499Z" }, +] + [[package]] name = "argcomplete" version = "3.6.3" @@ -890,6 +899,31 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/03/85/72020c6d9c185ea36742e005d9e8f3b47ff8a4181ced8ff03d66e655de6b/cyclopts-4.15.0-py3-none-any.whl", hash = "sha256:0eb784abb4ea8893099429e903193ff31edcb5fe17a619fa93c6eaf60bf51f1f", size = 215341, upload-time = "2026-05-21T12:23:23.549Z" }, ] +[[package]] +name = "datasets" +version = "5.0.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "dill" }, + { name = "filelock" }, + { name = "fsspec", extra = ["http"] }, + { name = "httpx" }, + { name = "huggingface-hub" }, + { name = "multiprocess" }, + { name = "numpy" }, + { name = "packaging" }, + { name = "pandas" }, + { name = "pyarrow" }, + { name = "pyyaml" }, + { name = "requests" }, + { name = "tqdm" }, + { name = "xxhash" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/d9/85/ce4f780c32f7e36d71257f1c27e8ba898ebe379cb54f211f5f2013f2c219/datasets-5.0.0.tar.gz", hash = "sha256:83dbbbdb07a33b82192b8c419deb18739b138ee2ce1a322d55ce6b100954ec1a", size = 631708, upload-time = "2026-06-05T13:18:26.124Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/05/66/73034ad30b59f13439b75e620989dacba4c047256e358ba7c2e9ec98ea22/datasets-5.0.0-py3-none-any.whl", hash = "sha256:7dd34927a0fd7046e98aad5cb9430e699c373238a15befa7b9bf22b991a7fee6", size = 555084, upload-time = "2026-06-05T13:18:24.435Z" }, +] + [[package]] name = "defusedxml" version = "0.7.1" @@ -899,6 +933,24 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/07/6c/aa3f2f849e01cb6a001cd8554a88d4c77c5c1a31c95bdf1cf9301e6d9ef4/defusedxml-0.7.1-py2.py3-none-any.whl", hash = "sha256:a352e7e428770286cc899e2542b6cdaedb2b4953ff269a210103ec58f6198a61", size = 25604, upload-time = "2021-03-08T10:59:24.45Z" }, ] +[[package]] +name = "dill" +version = "0.4.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/81/e1/56027a71e31b02ddc53c7d65b01e68edf64dea2932122fe7746a516f75d5/dill-0.4.1.tar.gz", hash = "sha256:423092df4182177d4d8ba8290c8a5b640c66ab35ec7da59ccfa00f6fa3eea5fa", size = 187315, upload-time = "2026-01-19T02:36:56.85Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1e/77/dc8c558f7593132cf8fefec57c4f60c83b16941c574ac5f619abb3ae7933/dill-0.4.1-py3-none-any.whl", hash = "sha256:1e1ce33e978ae97fcfcff5638477032b801c46c7c65cf717f95fbc2248f79a9d", size = 120019, upload-time = "2026-01-19T02:36:55.663Z" }, +] + +[[package]] +name = "diskcache" +version = "5.6.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/3f/21/1c1ffc1a039ddcc459db43cc108658f32c57d271d7289a2794e401d0fdb6/diskcache-5.6.3.tar.gz", hash = "sha256:2c3a3fa2743d8535d832ec61c2054a1641f41775aa7c556758a109941e33e4fc", size = 67916, upload-time = "2023-08-31T06:12:00.316Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3f/27/4570e78fc0bf5ea0ca45eb1de3818a23787af9b390c0b0a0033a1b8236f9/diskcache-5.6.3-py3-none-any.whl", hash = "sha256:5e31b2d5fbad117cc363ebaf6b689474db18a1f6438bc82358b024abd4c2ca19", size = 45550, upload-time = "2023-08-31T06:11:58.822Z" }, +] + [[package]] name = "distlib" version = "0.4.0" @@ -1148,7 +1200,7 @@ wheels = [ [[package]] name = "fireflyframework-agentic" -version = "26.6.13" +version = "26.6.14" source = { editable = "." } dependencies = [ { name = "genai-prices" }, @@ -1223,6 +1275,13 @@ dev = [ embeddings = [ { name = "numpy" }, ] +evaluation = [ + { name = "langchain-anthropic" }, + { name = "langchain-ollama" }, + { name = "langchain-openai" }, + { name = "numpy" }, + { name = "ragas" }, +] google-embeddings = [ { name = "google-generativeai" }, ] @@ -1292,10 +1351,14 @@ requires-dist = [ { name = "httpx", marker = "extra == 'dev'", specifier = ">=0.28.0" }, { name = "httpx", marker = "extra == 'ollama-embeddings'", specifier = ">=0.28.0" }, { name = "jinja2", specifier = ">=3.1.0" }, + { name = "langchain-anthropic", marker = "extra == 'evaluation'", specifier = ">=0.3" }, + { name = "langchain-ollama", marker = "extra == 'evaluation'", specifier = ">=0.3" }, + { name = "langchain-openai", marker = "extra == 'evaluation'", specifier = ">=0.3" }, { name = "markdown-it-py", specifier = ">=3.0" }, { name = "mistralai", marker = "extra == 'mistral-embeddings'", specifier = ">=1.0.0" }, { name = "motor", marker = "extra == 'mongodb'", specifier = ">=3.6.0" }, { name = "numpy", marker = "extra == 'embeddings'", specifier = ">=1.26.0" }, + { name = "numpy", marker = "extra == 'evaluation'", specifier = ">=1.26.0" }, { name = "numpy", marker = "extra == 'reasoning-eval'", specifier = ">=2.0.0" }, { name = "openai", marker = "extra == 'azure-embeddings'", specifier = ">=1.0.0" }, { name = "openai", marker = "extra == 'openai-embeddings'", specifier = ">=1.0.0" }, @@ -1321,6 +1384,7 @@ requires-dist = [ { name = "pytest-cov", marker = "extra == 'dev'", specifier = ">=6.0.0" }, { name = "python-dotenv", specifier = ">=1.0.0" }, { name = "qdrant-client", marker = "extra == 'vectorstores-qdrant'", specifier = ">=1.12.0" }, + { name = "ragas", marker = "extra == 'evaluation'", specifier = ">=0.2" }, { name = "ruff", marker = "extra == 'dev'", specifier = ">=0.9.0" }, { name = "sqlalchemy", marker = "extra == 'postgres'", specifier = ">=2.0.0" }, { name = "sqlite-vec", marker = "extra == 'vectorstores-sqlite-vec'", specifier = ">=0.1.6" }, @@ -1328,7 +1392,7 @@ requires-dist = [ { name = "voyageai", marker = "extra == 'voyage-embeddings'", specifier = ">=0.3.0" }, { name = "watchfiles", marker = "extra == 'watch'", specifier = ">=0.24.0" }, ] -provides-extras = ["postgres", "mongodb", "security", "script-execution", "embeddings", "openai-embeddings", "cohere-embeddings", "google-embeddings", "mistral-embeddings", "voyage-embeddings", "azure-embeddings", "bedrock-embeddings", "ollama-embeddings", "reasoning-eval", "vectorstores-chroma", "vectorstores-sqlite-vec", "vectorstores-pinecone", "vectorstores-qdrant", "vectorstores-pgvector", "watch", "binary", "all", "dev"] +provides-extras = ["postgres", "mongodb", "security", "script-execution", "embeddings", "openai-embeddings", "cohere-embeddings", "google-embeddings", "mistral-embeddings", "voyage-embeddings", "azure-embeddings", "bedrock-embeddings", "ollama-embeddings", "reasoning-eval", "vectorstores-chroma", "vectorstores-sqlite-vec", "vectorstores-pinecone", "vectorstores-qdrant", "vectorstores-pgvector", "watch", "binary", "all", "evaluation", "dev"] [[package]] name = "flatbuffers" @@ -1420,6 +1484,11 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d5/0c/043d5e551459da400957a1395e0febbf771446ff34291afcbe3d8be2a279/fsspec-2026.4.0-py3-none-any.whl", hash = "sha256:11ef7bb35dab8a394fde6e608221d5cf3e8499401c249bebaeaad760a1a8dec2", size = 203402, upload-time = "2026-04-29T20:42:36.842Z" }, ] +[package.optional-dependencies] +http = [ + { name = "aiohttp" }, +] + [[package]] name = "future" version = "1.0.0" @@ -1977,6 +2046,28 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" }, ] +[[package]] +name = "instructor" +version = "1.15.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "aiohttp" }, + { name = "docstring-parser" }, + { name = "jinja2" }, + { name = "jiter" }, + { name = "openai" }, + { name = "pydantic" }, + { name = "pydantic-core" }, + { name = "requests" }, + { name = "rich" }, + { name = "tenacity" }, + { name = "typer" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/9f/24/f6b28e83b3194c6223ed7c6eed5724687f6ecd378ec2ff24044f0cbf1f09/instructor-1.15.4.tar.gz", hash = "sha256:ea2280c3678d0f6891c4d826104f95624b680e69877113a6345b1d7c9027ba0f", size = 70049678, upload-time = "2026-06-28T07:36:43.458Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/80/8d/f668a30fff4d25b36533355e23aeb0b5724df4628eb974124ed64b7bcf8d/instructor-1.15.4-py3-none-any.whl", hash = "sha256:00e0ecda80fd9746fb6d082d3f9641e193adb1d8849f0775f91519a82aeff968", size = 252522, upload-time = "2026-06-28T07:36:36.863Z" }, +] + [[package]] name = "jaraco-classes" version = "3.4.0" @@ -2033,56 +2124,56 @@ wheels = [ [[package]] name = "jiter" -version = "0.15.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/66/b5/55f06bb281d92fb3cc86d14e1def2bd908bb77693183e7cb1f5a3c388b0c/jiter-0.15.0.tar.gz", hash = "sha256:4251acc80e2b7c9b7b8823456ea0fceeb0734dac2df7636d3c711b38476b5a76", size = 166640, upload-time = "2026-05-19T10:09:48.361Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/e5/f4/f708c900ecee41b2025ef8413d5351e5649eb2125c506f6720cc69b06f5c/jiter-0.15.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:1c11465f97e2abf45a014b83b730222f8f1c5335e802c7055a67d50de6f1f4e3", size = 307829, upload-time = "2026-05-19T10:07:59.704Z" }, - { url = "https://files.pythonhosted.org/packages/86/59/db537c0949e83668c38481d426b9f2fd5ab758c4ee53a811dd0a510626a0/jiter-0.15.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:d1e7b1776f0797956c509e123d0952d10d293a9492dea9f288ab9570ec01d1a5", size = 308445, upload-time = "2026-05-19T10:08:01.184Z" }, - { url = "https://files.pythonhosted.org/packages/37/38/ea0e13b18c30ef951da0d47d39e7fa9edb82a93a62990ffbd7cea9b622d4/jiter-0.15.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:351a341c2105aa430b7047e30f1bf7975f6313b00165d3fc07be2edaf741f279", size = 336181, upload-time = "2026-05-19T10:08:02.688Z" }, - { url = "https://files.pythonhosted.org/packages/58/fc/2303901b16c4ba05865588990a420c0b4156270b44379c20931544a1d962/jiter-0.15.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:4ab395feec8d249ec4044e228e98a7033f043426a265df439dc3698823f0a4e4", size = 362985, upload-time = "2026-05-19T10:08:04.394Z" }, - { url = "https://files.pythonhosted.org/packages/5b/6f/11bace093c52e7d4d26c8e606ccd7ae8c972189622469ec0d9e28161e28b/jiter-0.15.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:a2a438005b6f22d0273413484d6094d7c2c5d10ec1b3a3bf128e0d1d3ba53258", size = 453292, upload-time = "2026-05-19T10:08:05.967Z" }, - { url = "https://files.pythonhosted.org/packages/22/db/987f2f086ca4d7a6582eb4ccd513f9b26b42d9e4243a087609a3137a8fc7/jiter-0.15.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:f18f85e4218d1b40f000f42a92239a7a61a902cd42c65e6c360dbd17dcb20894", size = 373501, upload-time = "2026-05-19T10:08:07.857Z" }, - { url = "https://files.pythonhosted.org/packages/8f/7c/89fbcabb2739b7a5b8dc959a1b6c5761f6484f5fed3486854b3c789bb1de/jiter-0.15.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d1aa62e277fc1cbd80e6deacae6f4d983b41b3d7728e0645c5d741a6149bba45", size = 344683, upload-time = "2026-05-19T10:08:09.431Z" }, - { url = "https://files.pythonhosted.org/packages/30/6f/6cca7692e7dddfec6d8d76c54dc97f2af2a41df4ac0674b999df1f09a5f3/jiter-0.15.0-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:6550fa135c7deb8ead6af49ed7ff648532ea8334a1447fe34a36315ef79c5c29", size = 350892, upload-time = "2026-05-19T10:08:11.352Z" }, - { url = "https://files.pythonhosted.org/packages/39/14/0338d6190cb8e6d22e677ab1d4eabd4117f67cca70c54cd04b82ff64e068/jiter-0.15.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:066f8f33f18b2419cd8213b2436fa7fbc9c499f315971cfa3ce1f9820c001b1b", size = 388723, upload-time = "2026-05-19T10:08:12.912Z" }, - { url = "https://files.pythonhosted.org/packages/90/31/cc19f4a1bdb6afb09ce6a2f2615aa8d44d994eba0d8e6105ed1af920e736/jiter-0.15.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:75e8a04e91432dde9f1838373cf93d23726c79d3e908d319acf0e796f85592e7", size = 516648, upload-time = "2026-05-19T10:08:14.808Z" }, - { url = "https://files.pythonhosted.org/packages/49/9f/833c541512cd091b63c10c0381973dfe11bc7a503a818c16384417e0c81e/jiter-0.15.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:a97261f1fccb8e50ecd2890a96e46efdc3f57c80a197324c6777827231eca712", size = 547382, upload-time = "2026-05-19T10:08:16.927Z" }, - { url = "https://files.pythonhosted.org/packages/d2/11/e7b70e91f90bc4477e8eee9e8a5f7cf3cb41b4525d6394dc98a714eb8f7f/jiter-0.15.0-cp313-cp313-win32.whl", hash = "sha256:c77496cb10bd7549690fbbab3e5ec05857b83e49276f4a9423a766ddd2afcd4c", size = 205845, upload-time = "2026-05-19T10:08:18.401Z" }, - { url = "https://files.pythonhosted.org/packages/4b/23/5c20d9ad6f02c493e4023e5d2d09e1c1f15fe2753c9102c544aff068a88e/jiter-0.15.0-cp313-cp313-win_amd64.whl", hash = "sha256:b15741f501469009ae0ae90b7147958a664a7dede40aa7ff174a8a4645f546d0", size = 196842, upload-time = "2026-05-19T10:08:20.131Z" }, - { url = "https://files.pythonhosted.org/packages/6b/11/1eb400ef248e8c925fd883fbe325daf5e42cd1b0d308539dd332bd4f7ffc/jiter-0.15.0-cp313-cp313-win_arm64.whl", hash = "sha256:5d6a60072b44c3c2b797a7ddcbcbbf2b34ea3cfd4721580fbfd2a09d9d9b84ba", size = 192212, upload-time = "2026-05-19T10:08:21.807Z" }, - { url = "https://files.pythonhosted.org/packages/8a/60/2fd8d7c79da8acf9b7b277c7616847773779356b92acfc9bb158452174da/jiter-0.15.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:ef1fd24d9413f6209e00d3d5a453e67acfe004a25cc6c8e8484faed4311ab9e8", size = 315065, upload-time = "2026-05-19T10:08:23.218Z" }, - { url = "https://files.pythonhosted.org/packages/46/f4/008fb7d65e8ac2abf00811651a661e025c4ba80bbc6f378450384ddd3aed/jiter-0.15.0-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:144f8e72cb53dab146347b91cceac01f5481237f2b93b4a339a1ee8f8878b67c", size = 339444, upload-time = "2026-05-19T10:08:24.701Z" }, - { url = "https://files.pythonhosted.org/packages/00/55/90b0c7b9c6896c0f2a591dd36d36b71d22e09674bfef178fa03ba3f81499/jiter-0.15.0-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:553fcac2ef2cb990877f9fc0833b8b629a3e6a5670b6b5fd58219b41a653ddc4", size = 347779, upload-time = "2026-05-19T10:08:26.408Z" }, - { url = "https://files.pythonhosted.org/packages/51/6b/69666cec5000fd57734c118437394516c749ae8dbeea9fb66d6fef9c4775/jiter-0.15.0-cp313-cp313t-win_amd64.whl", hash = "sha256:774f93f65031856bf14ad9f59bdcab8b8cad501e5ceabd51ba3525f76937a25b", size = 200395, upload-time = "2026-05-19T10:08:28.055Z" }, - { url = "https://files.pythonhosted.org/packages/39/04/a6aa62cd27e8149b0d28df5561f10f6cceaf7935a9ccf3f1c5a05f9a0cd8/jiter-0.15.0-cp313-cp313t-win_arm64.whl", hash = "sha256:f1e1754960f38ec40613a07e5e372df67acb3b890fb383b6fb3de3e49ddbf3c7", size = 190516, upload-time = "2026-05-19T10:08:29.35Z" }, - { url = "https://files.pythonhosted.org/packages/eb/d2/079f350ebf7859d081de30aa890f9e3be68516f754f3ba32366ffff4dcee/jiter-0.15.0-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:ac0d9ddea4350974be7a221fc25895f251a8fee748c889bdced2141c0fec1a49", size = 308884, upload-time = "2026-05-19T10:08:31.667Z" }, - { url = "https://files.pythonhosted.org/packages/04/4e/a2c30a7f69b48c03b20935d647479106fe932f6e63f75faf53937197e05d/jiter-0.15.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:01a8222cf05ab1128e239421156c207949808acaaea2bdfd33130ae666786e86", size = 310028, upload-time = "2026-05-19T10:08:33.304Z" }, - { url = "https://files.pythonhosted.org/packages/40/90/2e7cdfd3cf8ca967be38c48f5cf474d79f089efaf559a40f15984a77ae69/jiter-0.15.0-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:182226cbc930c9fab81bc2e41a4da672f89539906dadb05e75670ac07b94f71f", size = 337485, upload-time = "2026-05-19T10:08:35.259Z" }, - { url = "https://files.pythonhosted.org/packages/9b/11/15a1aa28b120b8ee5b4f1fb894c125046225f09847738bd64233d3b84883/jiter-0.15.0-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:71683c38c825452999b5717fcae07ea708e8c93003e808be4319c1b02e3d176e", size = 364223, upload-time = "2026-05-19T10:08:36.694Z" }, - { url = "https://files.pythonhosted.org/packages/b7/25/f442e8af5f3d0dcf47b39e83a0efd9ee45ea946aa6d04625dc3181eae3b6/jiter-0.15.0-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:30f2218e6a9e5c18bc10fe6d41ac189c442c88eacf11bad9f28ef95a9bef00e6", size = 456387, upload-time = "2026-05-19T10:08:38.143Z" }, - { url = "https://files.pythonhosted.org/packages/da/f4/37f2d2c9f64f49af7da652ed7532bb5a2372e588e6927c3fdd76f911db65/jiter-0.15.0-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:5157de9f76eb4bc5ea74a1219366a25f945ad305641d74e04f59c54087091aa9", size = 374461, upload-time = "2026-05-19T10:08:39.869Z" }, - { url = "https://files.pythonhosted.org/packages/60/28/edcfbbbf0cb15436f36664a8908a0df47ab9006298d4cd937dc08ea932d6/jiter-0.15.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:90c5db5527c221249a876160663ab891ace358c17f7b9c93ec1478b7f0550e5c", size = 345924, upload-time = "2026-05-19T10:08:41.668Z" }, - { url = "https://files.pythonhosted.org/packages/47/13/89fba6398dab7f202b7278c4b4aac122399d2c0183971c4a57a3b7088df5/jiter-0.15.0-cp314-cp314-manylinux_2_31_riscv64.whl", hash = "sha256:3e4540b8e74e4268811ac05db226a6a128ff572e7e0ce3f1163b693cadb184cd", size = 352283, upload-time = "2026-05-19T10:08:43.091Z" }, - { url = "https://files.pythonhosted.org/packages/1b/da/0f6af8cef2c565a1ab44d970f268c43ccaa72707386ea6388e6fe2b6cd26/jiter-0.15.0-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:62ebd14e47e9aed9df4472afcb2663668ce4d74891cd54f86bf6e44029d6dc89", size = 389985, upload-time = "2026-05-19T10:08:44.915Z" }, - { url = "https://files.pythonhosted.org/packages/a1/ec/b9cb7d6d29e24ee14910266157d2a279d7a8f60ee0df7fa840882976ba64/jiter-0.15.0-cp314-cp314-musllinux_1_1_aarch64.whl", hash = "sha256:0be6f5ad41a809f303f416d17cec92a7a725902fb9b4f3de3d19362ac0ef8554", size = 517695, upload-time = "2026-05-19T10:08:46.486Z" }, - { url = "https://files.pythonhosted.org/packages/64/5e/6d1bda880723aae0ad86b4b763f044362448efe31e3e819635d41cb03451/jiter-0.15.0-cp314-cp314-musllinux_1_1_x86_64.whl", hash = "sha256:813dfbb17d65328bf86e5f0905dd277ba2265d3ca20556e86c0c7035b7182e5a", size = 548868, upload-time = "2026-05-19T10:08:48.026Z" }, - { url = "https://files.pythonhosted.org/packages/0c/72/7de501cf38dcacaf35098796f3a50e0f2e338baba18a58946c618544b809/jiter-0.15.0-cp314-cp314-win32.whl", hash = "sha256:50e51156192722a9c58db112837d3f8ef96fb3c5ecc14e95f409134b08b158ec", size = 206380, upload-time = "2026-05-19T10:08:49.738Z" }, - { url = "https://files.pythonhosted.org/packages/1e/a9/e19addf4b0c1bdce52c6da12351e6bc42c340c45e7c09e2158e46d293ccc/jiter-0.15.0-cp314-cp314-win_amd64.whl", hash = "sha256:30ce1a5d16b5641dc935d50ef775af6a0871e3d14ab05d6fc54dff371b78e558", size = 197687, upload-time = "2026-05-19T10:08:51.088Z" }, - { url = "https://files.pythonhosted.org/packages/f2/c9/776b1db01db25fc6c1d58d1979a37b0a9fe787e5f5b1d062d2eaacb77923/jiter-0.15.0-cp314-cp314-win_arm64.whl", hash = "sha256:510c8b3c17a0ed9ac69850c0438dada3c9b82d9c4d589fcb62002a5a9cf3a866", size = 192571, upload-time = "2026-05-19T10:08:52.451Z" }, - { url = "https://files.pythonhosted.org/packages/a0/f6/45bb4670bacf300fd2c7abadbfb3af376e5f1b6ae75fd9bc069891d15870/jiter-0.15.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7553333dd0930c104a5a0db8df72bf7219fe663d731383b576bb6ed6351c984d", size = 317151, upload-time = "2026-05-19T10:08:53.867Z" }, - { url = "https://files.pythonhosted.org/packages/d7/68/ed635ad5acd7b73e454283083bbb7c8205ad10e88b0d9d7d793b09fe8226/jiter-0.15.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f2143ab06181d2b029eedcb6af3cebe95f11bbac62441781860f98ee9330a6a6", size = 341243, upload-time = "2026-05-19T10:08:55.383Z" }, - { url = "https://files.pythonhosted.org/packages/5d/db/3ff4176b817b8ea33879e71e13d8bc2b0d481a7ed3fe9e080f333d415c16/jiter-0.15.0-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:6eac374c5c975709b69c10f09afd199df74150172156ad10c8d4fd785b7da995", size = 363629, upload-time = "2026-05-19T10:08:56.928Z" }, - { url = "https://files.pythonhosted.org/packages/ab/24/5f8270e0ba9c883582f96f722f8a0b58015c7ce1f8c6d4571cf394e99b6b/jiter-0.15.0-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:b3b3b775e33d3bfaec9899edc526ae97b0da0bf9d071a46124ba419149a414f8", size = 456198, upload-time = "2026-05-19T10:08:58.618Z" }, - { url = "https://files.pythonhosted.org/packages/45/5b/76fc02b0b5c54c3d18c60653156e2f76fde1816f9b4722db68d6ee2c897e/jiter-0.15.0-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:eda3071db3346334beae1360b46da4606da57bf3528c167b3c38533afaf9f2c5", size = 373710, upload-time = "2026-05-19T10:09:00.151Z" }, - { url = "https://files.pythonhosted.org/packages/c4/52/4310821b0ea9277994d3e1f49fc6a4b34e4800caebacb2c0af81da59a454/jiter-0.15.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c6694a173ecabc12eb60efbc0b474464ead1951ff65cd8b1e72100715c64512b", size = 349901, upload-time = "2026-05-19T10:09:01.621Z" }, - { url = "https://files.pythonhosted.org/packages/93/fe/67648c35b3594fba8854ac64cc8a826d8bcd18324bbdb53d77697c60b6ef/jiter-0.15.0-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:a254e10b593624d230c365b6d616b22ca0ad65e63a16e6631c2b3466022e6ba8", size = 352438, upload-time = "2026-05-19T10:09:03.216Z" }, - { url = "https://files.pythonhosted.org/packages/cb/28/0a1879d07ad6b3e025a2750027363452ced93c2d16d1c9d4b153ffd51c91/jiter-0.15.0-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:d8d2955167274e15d79a7a020afdd9b39c990eb80b2d89fca695d92dcfdd38ec", size = 388152, upload-time = "2026-05-19T10:09:04.741Z" }, - { url = "https://files.pythonhosted.org/packages/c1/78/46c6f6b56ba85c90021f4afd72ed42f691f8f84daacb5fe27277070e3858/jiter-0.15.0-cp314-cp314t-musllinux_1_1_aarch64.whl", hash = "sha256:acf4ee4d1fc55917239fe72972fb292dd773055d05eb040d36f4326e02cc2c0e", size = 517707, upload-time = "2026-05-19T10:09:06.231Z" }, - { url = "https://files.pythonhosted.org/packages/ca/cb/720662d4c88fcad606e826fef5424365527ba43ce4868a479aed8f8c507e/jiter-0.15.0-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:e7196e56f1cd69af1dbb07dff02dcfb260a50b45a82d409d92a06fedb32473b5", size = 548241, upload-time = "2026-05-19T10:09:08.093Z" }, - { url = "https://files.pythonhosted.org/packages/60/e3/935b8034fd143f21125c87d51404a9e0e1449186a494405721ff5d1d695e/jiter-0.15.0-cp314-cp314t-win32.whl", hash = "sha256:7f6163c0f10b055245f814dcc59f4818da60dfe72f3e72ab89fc24b6bd5e9c52", size = 207950, upload-time = "2026-05-19T10:09:09.616Z" }, - { url = "https://files.pythonhosted.org/packages/93/59/984fd9ece895953dad3e0880a650e766f5a2da2c5514f0eafdaaabbeb5f9/jiter-0.15.0-cp314-cp314t-win_amd64.whl", hash = "sha256:980c256edb05b78a111b99c4de3b1d32e31634b867fd1fc2cf726e7b7bba9854", size = 200055, upload-time = "2026-05-19T10:09:11.367Z" }, - { url = "https://files.pythonhosted.org/packages/0e/a4/cf8d779feb133a27a2e3bc833bccb9e13aa332cdf820497ebf72c10ce8c3/jiter-0.15.0-cp314-cp314t-win_arm64.whl", hash = "sha256:66b1880df2d01e206e8339769d1c7c1753bcb653efd6289e203f6f24ebada0c0", size = 191244, upload-time = "2026-05-19T10:09:12.74Z" }, +version = "0.14.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/6e/c1/0cddc6eb17d4c53a99840953f95dd3accdc5cfc7a337b0e9b26476276be9/jiter-0.14.0.tar.gz", hash = "sha256:e8a39e66dac7153cf3f964a12aad515afa8d74938ec5cc0018adcdae5367c79e", size = 165725, upload-time = "2026-04-10T14:28:42.01Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/97/2a/09f70020898507a89279659a1afe3364d57fc1b2c89949081975d135f6f5/jiter-0.14.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:af72f204cf4d44258e5b4c1745130ac45ddab0e71a06333b01de660ab4187a94", size = 315502, upload-time = "2026-04-10T14:26:47.697Z" }, + { url = "https://files.pythonhosted.org/packages/d6/be/080c96a45cd74f9fce5db4fd68510b88087fb37ffe2541ff73c12db92535/jiter-0.14.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:4b77da71f6e819be5fbcec11a453fde5b1d0267ef6ed487e2a392fd8e14e4e3a", size = 314870, upload-time = "2026-04-10T14:26:49.149Z" }, + { url = "https://files.pythonhosted.org/packages/7d/5e/2d0fee155826a968a832cc32438de5e2a193292c8721ca70d0b53e58245b/jiter-0.14.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:77f4ea612fe8b84b8b04e51d0e78029ecf3466348e25973f953de6e6a59aa4c1", size = 343406, upload-time = "2026-04-10T14:26:50.762Z" }, + { url = "https://files.pythonhosted.org/packages/70/af/bf9ee0d3a4f8dc0d679fc1337f874fe60cdbf841ebbb304b374e1c9aaceb/jiter-0.14.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:62fe2451f8fcc0240261e6a4df18ecbcd58327857e61e625b2393ea3b468aac9", size = 369415, upload-time = "2026-04-10T14:26:52.188Z" }, + { url = "https://files.pythonhosted.org/packages/0f/83/8e8561eadba31f4d3948a5b712fb0447ec71c3560b57a855449e7b8ddc98/jiter-0.14.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:6112f26f5afc75bcb475787d29da3aa92f9d09c7858f632f4be6ffe607be82e9", size = 461456, upload-time = "2026-04-10T14:26:53.611Z" }, + { url = "https://files.pythonhosted.org/packages/f6/c9/c5299e826a5fe6108d172b344033f61c69b1bb979dd8d9ddd4278a160971/jiter-0.14.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:215a6cb8fb7dc702aa35d475cc00ddc7f970e5c0b1417fb4b4ac5d82fa2a29db", size = 378488, upload-time = "2026-04-10T14:26:55.211Z" }, + { url = "https://files.pythonhosted.org/packages/5d/37/c16d9d15c0a471b8644b1abe3c82668092a707d9bedcf076f24ff2e380cd/jiter-0.14.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fc4ab96a30fb3cb2c7e0cd33f7616c8860da5f5674438988a54ac717caccdbaa", size = 353242, upload-time = "2026-04-10T14:26:56.705Z" }, + { url = "https://files.pythonhosted.org/packages/58/ea/8050cb0dc654e728e1bfacbc0c640772f2181af5dedd13ae70145743a439/jiter-0.14.0-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:3a99c1387b1f2928f799a9de899193484d66206a50e98233b6b088a7f0c1edb2", size = 356823, upload-time = "2026-04-10T14:26:58.281Z" }, + { url = "https://files.pythonhosted.org/packages/b0/3b/cf71506d270e5f84d97326bf220e47aed9b95e9a4a060758fb07772170ab/jiter-0.14.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:ab18d11074485438695f8d34a1b6da61db9754248f96d51341956607a8f39985", size = 392564, upload-time = "2026-04-10T14:27:00.018Z" }, + { url = "https://files.pythonhosted.org/packages/b0/cc/8c6c74a3efb5bd671bfd14f51e8a73375464ca914b1551bc3b40e26ac2c9/jiter-0.14.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:801028dcfc26ac0895e4964cbc0fd62c73be9fd4a7d7b1aaf6e5790033a719b7", size = 520322, upload-time = "2026-04-10T14:27:01.664Z" }, + { url = "https://files.pythonhosted.org/packages/41/24/68d7b883ec959884ddf00d019b2e0e82ba81b167e1253684fa90519ce33c/jiter-0.14.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:ad425b087aafb4a1c7e1e98a279200743b9aaf30c3e0ba723aec93f061bd9bc8", size = 552619, upload-time = "2026-04-10T14:27:03.316Z" }, + { url = "https://files.pythonhosted.org/packages/b6/89/b1a0985223bbf3150ff9e8f46f98fc9360c1de94f48abe271bbe1b465682/jiter-0.14.0-cp313-cp313-win32.whl", hash = "sha256:882bcb9b334318e233950b8be366fe5f92c86b66a7e449e76975dfd6d776a01f", size = 205699, upload-time = "2026-04-10T14:27:04.662Z" }, + { url = "https://files.pythonhosted.org/packages/4c/19/3f339a5a7f14a11730e67f6be34f9d5105751d547b615ef593fa122a5ded/jiter-0.14.0-cp313-cp313-win_amd64.whl", hash = "sha256:9b8c571a5dba09b98bd3462b5a53f27209a5cbbe85670391692ede71974e979f", size = 201323, upload-time = "2026-04-10T14:27:06.139Z" }, + { url = "https://files.pythonhosted.org/packages/50/56/752dd89c84be0e022a8ea3720bcfa0a8431db79a962578544812ce061739/jiter-0.14.0-cp313-cp313-win_arm64.whl", hash = "sha256:34f19dcc35cb1abe7c369b3756babf8c7f04595c0807a848df8f26ef8298ef92", size = 191099, upload-time = "2026-04-10T14:27:07.564Z" }, + { url = "https://files.pythonhosted.org/packages/91/28/292916f354f25a1fe8cf2c918d1415c699a4a659ae00be0430e1c5d9ffea/jiter-0.14.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:e89bcd7d426a75bb4952c696b267075790d854a07aad4c9894551a82c5b574ab", size = 320880, upload-time = "2026-04-10T14:27:09.326Z" }, + { url = "https://files.pythonhosted.org/packages/ad/c7/b002a7d8b8957ac3d469bd59c18ef4b1595a5216ae0de639a287b9816023/jiter-0.14.0-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7b25beaa0d4447ea8c7ae0c18c688905d34840d7d0b937f2f7bdd52162c98a40", size = 346563, upload-time = "2026-04-10T14:27:11.287Z" }, + { url = "https://files.pythonhosted.org/packages/f9/3b/f8d07580d8706021d255a6356b8fab13ee4c869412995550ce6ed4ddf97d/jiter-0.14.0-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:651a8758dd413c51e3b7f6557cdc6921faf70b14106f45f969f091f5cda990ea", size = 357928, upload-time = "2026-04-10T14:27:12.729Z" }, + { url = "https://files.pythonhosted.org/packages/47/5b/ac1a974da29e35507230383110ffec59998b290a8732585d04e19a9eb5ba/jiter-0.14.0-cp313-cp313t-win_amd64.whl", hash = "sha256:e1a7eead856a5038a8d291f1447176ab0b525c77a279a058121b5fccee257f6f", size = 203519, upload-time = "2026-04-10T14:27:14.125Z" }, + { url = "https://files.pythonhosted.org/packages/96/6d/9fc8433d667d2454271378a79747d8c76c10b51b482b454e6190e511f244/jiter-0.14.0-cp313-cp313t-win_arm64.whl", hash = "sha256:2e692633a12cda97e352fdcd1c4acc971b1c28707e1e33aeef782b0cbf051975", size = 190113, upload-time = "2026-04-10T14:27:16.638Z" }, + { url = "https://files.pythonhosted.org/packages/4f/1e/354ed92461b165bd581f9ef5150971a572c873ec3b68a916d5aa91da3cc2/jiter-0.14.0-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:6f396837fc7577871ca8c12edaf239ed9ccef3bbe39904ae9b8b63ce0a48b140", size = 315277, upload-time = "2026-04-10T14:27:18.109Z" }, + { url = "https://files.pythonhosted.org/packages/a6/95/8c7c7028aa8636ac21b7a55faef3e34215e6ed0cbf5ae58258427f621aa3/jiter-0.14.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:a4d50ea3d8ba4176f79754333bd35f1bbcd28e91adc13eb9b7ca91bc52a6cef9", size = 315923, upload-time = "2026-04-10T14:27:19.603Z" }, + { url = "https://files.pythonhosted.org/packages/47/40/e2a852a44c4a089f2681a16611b7ce113224a80fd8504c46d78491b47220/jiter-0.14.0-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ce17f8a050447d1b4153bda4fb7d26e6a9e74eb4f4a41913f30934c5075bf615", size = 344943, upload-time = "2026-04-10T14:27:21.262Z" }, + { url = "https://files.pythonhosted.org/packages/fc/1f/670f92adee1e9895eac41e8a4d623b6da68c4d46249d8b556b60b63f949e/jiter-0.14.0-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:f4f1c4b125e1652aefbc2e2c1617b60a160ab789d180e3d423c41439e5f32850", size = 369725, upload-time = "2026-04-10T14:27:22.766Z" }, + { url = "https://files.pythonhosted.org/packages/01/2f/541c9ba567d05de1c4874a0f8f8c5e3fd78e2b874266623da9a775cf46e0/jiter-0.14.0-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:be808176a6a3a14321d18c603f2d40741858a7c4fc982f83232842689fe86dd9", size = 461210, upload-time = "2026-04-10T14:27:24.315Z" }, + { url = "https://files.pythonhosted.org/packages/ce/a9/c31cbec09627e0d5de7aeaec7690dba03e090caa808fefd8133137cf45bc/jiter-0.14.0-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:26679d58ba816f88c3849306dd58cb863a90a1cf352cdd4ef67e30ccf8a77994", size = 380002, upload-time = "2026-04-10T14:27:26.155Z" }, + { url = "https://files.pythonhosted.org/packages/50/02/3c05c1666c41904a2f607475a73e7a4763d1cbde2d18229c4f85b22dc253/jiter-0.14.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:80381f5a19af8fa9aef743f080e34f6b25ebd89656475f8cf0470ec6157052aa", size = 354678, upload-time = "2026-04-10T14:27:27.701Z" }, + { url = "https://files.pythonhosted.org/packages/7d/97/e15b33545c2b13518f560d695f974b9891b311641bdcf178d63177e8801e/jiter-0.14.0-cp314-cp314-manylinux_2_31_riscv64.whl", hash = "sha256:004df5fdb8ecbd6d99f3227df18ba1a259254c4359736a2e6f036c944e02d7c5", size = 358920, upload-time = "2026-04-10T14:27:29.256Z" }, + { url = "https://files.pythonhosted.org/packages/ad/d2/8b1461def6b96ba44530df20d07ef7a1c7da22f3f9bf1727e2d611077bf1/jiter-0.14.0-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:cff5708f7ed0fa098f2b53446c6fa74c48469118e5cd7497b4f1cd569ab06928", size = 394512, upload-time = "2026-04-10T14:27:31.344Z" }, + { url = "https://files.pythonhosted.org/packages/e3/88/837566dd6ed6e452e8d3205355afd484ce44b2533edfa4ed73a298ea893e/jiter-0.14.0-cp314-cp314-musllinux_1_1_aarch64.whl", hash = "sha256:2492e5f06c36a976d25c7cc347a60e26d5470178d44cde1b9b75e60b4e519f28", size = 521120, upload-time = "2026-04-10T14:27:33.299Z" }, + { url = "https://files.pythonhosted.org/packages/89/6b/b00b45c4d1b4c031777fe161d620b755b5b02cdade1e316dcb46e4471d63/jiter-0.14.0-cp314-cp314-musllinux_1_1_x86_64.whl", hash = "sha256:7609cfbe3a03d37bfdbf5052012d5a879e72b83168a363deae7b3a26564d57de", size = 553668, upload-time = "2026-04-10T14:27:34.868Z" }, + { url = "https://files.pythonhosted.org/packages/ad/d8/6fe5b42011d19397433d345716eac16728ac241862a2aac9c91923c7509a/jiter-0.14.0-cp314-cp314-win32.whl", hash = "sha256:7282342d32e357543565286b6450378c3cd402eea333fc1ebe146f1fabb306fc", size = 207001, upload-time = "2026-04-10T14:27:36.455Z" }, + { url = "https://files.pythonhosted.org/packages/e5/43/5c2e08da1efad5e410f0eaaabeadd954812612c33fbbd8fd5328b489139d/jiter-0.14.0-cp314-cp314-win_amd64.whl", hash = "sha256:bd77945f38866a448e73b0b7637366afa814d4617790ecd88a18ca74377e6c02", size = 202187, upload-time = "2026-04-10T14:27:38Z" }, + { url = "https://files.pythonhosted.org/packages/aa/1f/6e39ac0b4cdfa23e606af5b245df5f9adaa76f35e0c5096790da430ca506/jiter-0.14.0-cp314-cp314-win_arm64.whl", hash = "sha256:f2d4c61da0821ee42e0cdf5489da60a6d074306313a377c2b35af464955a3611", size = 192257, upload-time = "2026-04-10T14:27:39.504Z" }, + { url = "https://files.pythonhosted.org/packages/05/57/7dbc0ffbbb5176a27e3518716608aa464aee2e2887dc938f0b900a120449/jiter-0.14.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:1bf7ff85517dd2f20a5750081d2b75083c1b269cf75afc7511bdf1f9548beb3b", size = 323441, upload-time = "2026-04-10T14:27:41.039Z" }, + { url = "https://files.pythonhosted.org/packages/83/6e/7b3314398d8983f06b557aa21b670511ec72d3b79a68ee5e4d9bff972286/jiter-0.14.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c8ef8791c3e78d6c6b157c6d360fbb5c715bebb8113bc6a9303c5caff012754a", size = 348109, upload-time = "2026-04-10T14:27:42.552Z" }, + { url = "https://files.pythonhosted.org/packages/ae/4f/8dc674bcd7db6dba566de73c08c763c337058baff1dbeb34567045b27cdc/jiter-0.14.0-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e74663b8b10da1fe0f4e4703fd7980d24ad17174b6bb35d8498d6e3ebce2ae6a", size = 368328, upload-time = "2026-04-10T14:27:44.574Z" }, + { url = "https://files.pythonhosted.org/packages/3b/5f/188e09a1f20906f98bbdec44ed820e19f4e8eb8aff88b9d1a5a497587ff3/jiter-0.14.0-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:1aca29ba52913f78362ec9c2da62f22cdc4c3083313403f90c15460979b84d9b", size = 463301, upload-time = "2026-04-10T14:27:46.717Z" }, + { url = "https://files.pythonhosted.org/packages/ac/f0/19046ef965ed8f349e8554775bb12ff4352f443fbe12b95d31f575891256/jiter-0.14.0-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:8b39b7d87a952b79949af5fef44d2544e58c21a28da7f1bae3ef166455c61746", size = 378891, upload-time = "2026-04-10T14:27:48.32Z" }, + { url = "https://files.pythonhosted.org/packages/c4/c3/da43bd8431ee175695777ee78cf0e93eacbb47393ff493f18c45231b427d/jiter-0.14.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:78d918a68b26e9fab068c2b5453577ef04943ab2807b9a6275df2a812599a310", size = 360749, upload-time = "2026-04-10T14:27:49.88Z" }, + { url = "https://files.pythonhosted.org/packages/72/26/e054771be889707c6161dbdec9c23d33a9ec70945395d70f07cfea1e9a6f/jiter-0.14.0-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:b08997c35aee1201c1a5361466a8fb9162d03ae7bf6568df70b6c859f1e654a4", size = 358526, upload-time = "2026-04-10T14:27:51.504Z" }, + { url = "https://files.pythonhosted.org/packages/c3/0f/7bea65ea2a6d91f2bf989ff11a18136644392bf2b0497a1fa50934c30a9c/jiter-0.14.0-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:260bf7ca20704d58d41f669e5e9fe7fe2fa72901a6b324e79056f5d52e9c9be2", size = 393926, upload-time = "2026-04-10T14:27:53.368Z" }, + { url = "https://files.pythonhosted.org/packages/3c/a1/b1ff7d70deef61ac0b7c6c2f12d2ace950cdeecb4fdc94500a0926802857/jiter-0.14.0-cp314-cp314t-musllinux_1_1_aarch64.whl", hash = "sha256:37826e3df29e60f30a382f9294348d0238ef127f4b5d7f5f8da78b5b9e050560", size = 521052, upload-time = "2026-04-10T14:27:55.058Z" }, + { url = "https://files.pythonhosted.org/packages/0b/7b/3b0649983cbaf15eda26a414b5b1982e910c67bd6f7b1b490f3cfc76896a/jiter-0.14.0-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:645be49c46f2900937ba0eaf871ad5183c96858c0af74b6becc7f4e367e36e06", size = 553716, upload-time = "2026-04-10T14:27:57.269Z" }, + { url = "https://files.pythonhosted.org/packages/97/f8/33d78c83bd93ae0c0af05293a6660f88a1977caef39a6d72a84afab94ce0/jiter-0.14.0-cp314-cp314t-win32.whl", hash = "sha256:2f7877ed45118de283786178eceaf877110abacd04fde31efff3940ae9672674", size = 207957, upload-time = "2026-04-10T14:27:59.285Z" }, + { url = "https://files.pythonhosted.org/packages/d6/ac/2b760516c03e2227826d1f7025d89bf6bf6357a28fe75c2a2800873c50bf/jiter-0.14.0-cp314-cp314t-win_amd64.whl", hash = "sha256:14c0cb10337c49f5eafe8e7364daca5e29a020ea03580b8f8e6c597fed4e1588", size = 204690, upload-time = "2026-04-10T14:28:00.962Z" }, + { url = "https://files.pythonhosted.org/packages/dc/2e/a44c20c58aeed0355f2d326969a181696aeb551a25195f47563908a815be/jiter-0.14.0-cp314-cp314t-win_arm64.whl", hash = "sha256:5419d4aa2024961da9fe12a9cfe7484996735dca99e8e090b5c88595ef1951ff", size = 191338, upload-time = "2026-04-10T14:28:02.853Z" }, ] [[package]] @@ -2225,9 +2316,77 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/cd/d2/6f99ca9c7eb961dfdd45b9643101399a8ee20922c662c362c91e9cc7e832/kubernetes-36.0.0-py2.py3-none-any.whl", hash = "sha256:a766433357ec9f90db7565cccf52e28e7fca40b0ef366c80a6022adbc0ac0425", size = 4660469, upload-time = "2026-05-20T20:44:20.893Z" }, ] +[[package]] +name = "langchain" +version = "1.3.11" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "langchain-core" }, + { name = "langgraph" }, + { name = "pydantic" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b9/a2/91a7197c604a3ce1b774b3c10dd114c3c745c6186a304fc2573b3f94d400/langchain-1.3.11.tar.gz", hash = "sha256:f3cf9cd4d2329b1a03eb8fd92b9d73e4e58a4d52570d67725fc77fbe0f104b32", size = 633374, upload-time = "2026-06-22T23:00:33.44Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/df/a4/3a181967294f8876362cc4ba36840d50b8286fa23bb3f5e602b69eb3cb1e/langchain-1.3.11-py3-none-any.whl", hash = "sha256:7ae011f95a09b22feea1e8ae4e43f0b6164aebf4c61b8ad845b45f72ff3a90a2", size = 133639, upload-time = "2026-06-22T23:00:31.619Z" }, +] + +[[package]] +name = "langchain-anthropic" +version = "1.4.8" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anthropic" }, + { name = "langchain-core" }, + { name = "pydantic" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/98/22/40ab129b08329ca295b391aa1d48267692b42594757084c6918e22b655ac/langchain_anthropic-1.4.8.tar.gz", hash = "sha256:c76891b2044d56105ff13c106ed12650637b53bd598a4bdf15b4796eefa2a4ec", size = 708524, upload-time = "2026-06-26T21:28:46.916Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b7/14/746235c4da89d9bc6a608c5f489f628e03feb8f697195c146e452c8f23c8/langchain_anthropic-1.4.8-py3-none-any.whl", hash = "sha256:778e9301b6fd517824f76ec1776975ce8add97a1f6a36c50ae3c2f4b03a66f7f", size = 52366, upload-time = "2026-06-26T21:28:45.535Z" }, +] + +[[package]] +name = "langchain-classic" +version = "1.0.8" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "langchain-core" }, + { name = "langchain-text-splitters" }, + { name = "langsmith" }, + { name = "pydantic" }, + { name = "pyyaml" }, + { name = "requests" }, + { name = "sqlalchemy" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/8d/65/6b5e8a7ff2f2968652c88a67dcecb925b9d8f0a0ce9458c76cd5a0dbd138/langchain_classic-1.0.8.tar.gz", hash = "sha256:ada0cc341a8a5b80fb24d73bdfaaeb849056ee2d8a41cc468355163fd3667484", size = 10557071, upload-time = "2026-06-10T21:27:54.866Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/99/9a/b8f5cb7490fdbf233088031fc69c9c747439d4097f67f196c1eb4869916d/langchain_classic-1.0.8-py3-none-any.whl", hash = "sha256:1a11ea7fbe630c4f2af2f3873d27718ceac9488cf32d0821030be7cf039a6213", size = 1041536, upload-time = "2026-06-10T21:27:52.767Z" }, +] + +[[package]] +name = "langchain-community" +version = "0.4.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "aiohttp" }, + { name = "httpx-sse" }, + { name = "langchain-classic" }, + { name = "langchain-core" }, + { name = "langsmith" }, + { name = "numpy" }, + { name = "pydantic-settings" }, + { name = "pyyaml" }, + { name = "requests" }, + { name = "sqlalchemy" }, + { name = "tenacity" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ea/0c/e3aca1f2b1c5b95f8b87cb2b6e81a6f20d538c07a128419dc01cef0617b6/langchain_community-0.4.2.tar.gz", hash = "sha256:a99308160d53d7e9b5965ee665e5173709914338210089fd5788ad724432c21e", size = 33268708, upload-time = "2026-05-22T19:42:59.374Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/8f/39/5d97e42a3e95dc2a6d71b2f902a3fae71786131e11d01bddb604accb0ebe/langchain_community-0.4.2-py3-none-any.whl", hash = "sha256:84dd8c5122532394d5b6849a5fc9995ef28e4f77227daeb09f24b3d942e9e466", size = 2364406, upload-time = "2026-05-22T19:42:57.103Z" }, +] + [[package]] name = "langchain-core" -version = "1.4.0" +version = "1.4.8" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "jsonpatch" }, @@ -2240,21 +2399,48 @@ dependencies = [ { name = "typing-extensions" }, { name = "uuid-utils" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/59/de/679a53472c25860837e32c0442c962fa86e95317a36460e2c9d5c91b17c2/langchain_core-1.4.0.tar.gz", hash = "sha256:1dc341eed802ed9c117c0df3923c991e5e9e226571e5725c194eeb5bd93d1a7f", size = 920260, upload-time = "2026-05-11T18:42:35.919Z" } +sdist = { url = "https://files.pythonhosted.org/packages/12/e3/bea6d0080acf183332f24dcd74c208aee5857cf8f783c3fb0bd86027d8fb/langchain_core-1.4.8.tar.gz", hash = "sha256:5bf1f8411077c904182ad8f975943d36adcbf579c4e017b3a118b719229ebf9a", size = 957974, upload-time = "2026-06-18T19:39:23.636Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/13/d6/bdf6f0481cc57ef300d6b1eb48cf1400c0409be715d6eb3cabadd1142a09/langchain_core-1.4.8-py3-none-any.whl", hash = "sha256:d84c28b05e3ba8d4271d0827aad5b592ccdaaf986e76768c23503f0a2045e8aa", size = 557416, upload-time = "2026-06-18T19:39:21.902Z" }, +] + +[[package]] +name = "langchain-ollama" +version = "1.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "langchain-core" }, + { name = "ollama" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/d4/9b/6641afe8a5bf807e454fd464eddfc7eb2f2df53cb0b29744381171f9c609/langchain_ollama-1.1.0.tar.gz", hash = "sha256:f776f56f6782ae4da7692579b94a6575906118318d1023b455d7207f9d059811", size = 133075, upload-time = "2026-04-07T02:48:00.873Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/0f/1a/86c38c27b81913a1c6c12448cab55defb5a1097c7dc9a4cea83f55477a2d/langchain_core-1.4.0-py3-none-any.whl", hash = "sha256:23cbbdb46e38ddd1dd5247e6167e96013eae74bea4c5949c550809970a9e565c", size = 548120, upload-time = "2026-05-11T18:42:33.992Z" }, + { url = "https://files.pythonhosted.org/packages/2c/b2/c2acb076590a98bee2816ed5f285e00df162a34238f9e276e175e14ebc35/langchain_ollama-1.1.0-py3-none-any.whl", hash = "sha256:43ac83a6eacb0f43855810739794dd55019e0d9b17bdcf3ecb3b1991ac3b59dd", size = 31413, upload-time = "2026-04-07T02:47:59.642Z" }, +] + +[[package]] +name = "langchain-openai" +version = "1.3.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "langchain-core" }, + { name = "openai" }, + { name = "tiktoken" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/d0/f3/b38e052f943a75ba0e6762c0a50b6f1d6bcd52a9ce63386d8803c2ff506e/langchain_openai-1.3.3.tar.gz", hash = "sha256:143769bf943820b80db769e47ca8fd0aac08ed18714519333b044c4431e9aa67", size = 3256559, upload-time = "2026-06-22T22:54:05.445Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2f/4b/7520114de1ce36bb17cbc98d0fcda048a9f755bedaeb73b92137aeaaf1db/langchain_openai-1.3.3-py3-none-any.whl", hash = "sha256:e469659862c8aabba4f6653df973206e7be54f98cf2275c86be7f06b7abe20d7", size = 120437, upload-time = "2026-06-22T22:54:03.8Z" }, ] [[package]] name = "langchain-protocol" -version = "0.0.15" +version = "0.0.18" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "typing-extensions" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/4f/24/9777489d6fbbee64af0c8f96d4f840239c408cf694f3394672807dafc490/langchain_protocol-0.0.15.tar.gz", hash = "sha256:9ab2d11ee73944754f10e037e717098d3a6796f0e58afa9cadda6154e7655ade", size = 5862, upload-time = "2026-05-01T22:30:04.748Z" } +sdist = { url = "https://files.pythonhosted.org/packages/d2/59/b5959aea96faa9146e2e49a7a22882b3528c62efafe9a6a95beab30c2305/langchain_protocol-0.0.18.tar.gz", hash = "sha256:ec3e11782f1ed0c9db38e5a9ed01b0e7a0d3fba406faa8aef6594b73c56a63e6", size = 6150, upload-time = "2026-06-18T17:08:26.959Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/1d/7a/9c97a7b9cbe4c5dc6a44cdb1545450c28f0c8ce89b9c1f0ee7fbad896263/langchain_protocol-0.0.15-py3-none-any.whl", hash = "sha256:461eb794358f83d5e42635a5797799ffec7b4702314e34edf73ac21e75d3ef79", size = 6982, upload-time = "2026-05-01T22:30:03.877Z" }, + { url = "https://files.pythonhosted.org/packages/99/2e/d82db9eec13ad0f72e7aaad5c4bc730ab111934fdc83c85523206eb9b0a0/langchain_protocol-0.0.18-py3-none-any.whl", hash = "sha256:70b53a86fbf9cedc863555effe44da192ab02d556ddbf2cf95b8873adcf41b5a", size = 7221, upload-time = "2026-06-18T17:08:25.996Z" }, ] [[package]] @@ -2269,6 +2455,65 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d3/26/1ef06f56198d631296d646a6223de35bcc6cf9795ceb2442816bc963b84c/langchain_text_splitters-1.1.2-py3-none-any.whl", hash = "sha256:a2de0d799ff31886429fd6e2e0032df275b60ec817c19059a7b46181cc1c2f10", size = 35903, upload-time = "2026-04-16T14:20:38.243Z" }, ] +[[package]] +name = "langgraph" +version = "1.2.7" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "langchain-core" }, + { name = "langgraph-checkpoint" }, + { name = "langgraph-prebuilt" }, + { name = "langgraph-sdk" }, + { name = "pydantic" }, + { name = "xxhash" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/bf/11/c3961537ac7577125aabba68effc33ecc3dc0962e2d287d734dbd61caad7/langgraph-1.2.7.tar.gz", hash = "sha256:dcdf5b441bf8c7c7c154e603b302c9dbfbfd2d11e1b7ae7d93a5aba979dc87bd", size = 721385, upload-time = "2026-06-30T01:24:12.528Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/16/5d/19de0c674cf2d8bc95b25cdb4c13ba3f4fbe0f35bf2224ef19cc46bbb4cd/langgraph-1.2.7-py3-none-any.whl", hash = "sha256:249349a5d6a32cb0aa2304a8fb6529bd0cf56db8780d21db9e086e5383668f89", size = 246930, upload-time = "2026-06-30T01:24:11.15Z" }, +] + +[[package]] +name = "langgraph-checkpoint" +version = "4.1.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "langchain-core" }, + { name = "ormsgpack" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/83/47/886af6f886f0bff2273164a45f008694e48a96ff3cd25ff0228f2aa9480e/langgraph_checkpoint-4.1.1.tar.gz", hash = "sha256:6c2bdb530c91f91d7d9c1bd100925d0fc4f498d418c17f3587d1526279482a25", size = 184020, upload-time = "2026-05-22T16:57:38.503Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/bd/b4/71425e3e38be92611300b9cc5e46a5bf98ab23f5ea8a75b73d02a2f1413c/langgraph_checkpoint-4.1.1-py3-none-any.whl", hash = "sha256:25d29144b082827218e7bc3f1e9b0566a4bb007895cd6cc26f66a8428739f56e", size = 56212, upload-time = "2026-05-22T16:57:37.203Z" }, +] + +[[package]] +name = "langgraph-prebuilt" +version = "1.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "langchain-core" }, + { name = "langgraph-checkpoint" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/29/66/ed9b93f56bc17ef22d551892f0ac2b225a97fe0fcf23a511b857f70d590b/langgraph_prebuilt-1.1.0.tar.gz", hash = "sha256:3c579cf6eed2d17f9c157c2d0fcaddcd8688524e7022d3b22b37a3bf4589d528", size = 178833, upload-time = "2026-05-12T03:37:49.332Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e9/43/3fe1a700b8490ed02679cdbbc8c915eb23a092faf496c9c1118abcd10be3/langgraph_prebuilt-1.1.0-py3-none-any.whl", hash = "sha256:51e311747d755b751d5c6b39b0c1446124d3a7643d2515017e6714b323508fc9", size = 41043, upload-time = "2026-05-12T03:37:48.007Z" }, +] + +[[package]] +name = "langgraph-sdk" +version = "0.4.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "httpx" }, + { name = "langchain-core" }, + { name = "langchain-protocol" }, + { name = "orjson" }, + { name = "websockets" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b4/2b/bd8ac26d4e97f6df88ef05ce5b6a38945a3903e1025d926f4752aa88aa97/langgraph_sdk-0.4.2.tar.gz", hash = "sha256:b88f0f5f6328ac0680d6790614a905b2bcfa257f2276dba4e38f0e86db0aa738", size = 348327, upload-time = "2026-06-01T17:51:19.856Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a0/05/aac507337cceae773c2cc9ab91eb6301963af7aeeb55b4217a00e15aff17/langgraph_sdk-0.4.2-py3-none-any.whl", hash = "sha256:75fa5096c1177ce39c847096a8fe3745ffd480ddb412995f836e9f5f884c43dd", size = 160521, upload-time = "2026-06-01T17:51:18.849Z" }, +] + [[package]] name = "langsmith" version = "0.8.5" @@ -2660,6 +2905,23 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/81/08/7036c080d7117f28a4af526d794aab6a84463126db031b007717c1a6676e/multidict-6.7.1-py3-none-any.whl", hash = "sha256:55d97cc6dae627efa6a6e548885712d4864b81110ac76fa4e534c03819fa4a56", size = 12319, upload-time = "2026-01-26T02:46:44.004Z" }, ] +[[package]] +name = "multiprocess" +version = "0.70.19" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "dill" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a2/f2/e783ac7f2aeeed14e9e12801f22529cc7e6b7ab80928d6dcce4e9f00922d/multiprocess-0.70.19.tar.gz", hash = "sha256:952021e0e6c55a4a9fe4cd787895b86e239a40e76802a789d6305398d3975897", size = 2079989, upload-time = "2026-01-19T06:47:39.744Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e3/45/8004d1e6b9185c1a444d6b55ac5682acf9d98035e54386d967366035a03a/multiprocess-0.70.19-py310-none-any.whl", hash = "sha256:97404393419dcb2a8385910864eedf47a3cadf82c66345b44f036420eb0b5d87", size = 134948, upload-time = "2026-01-19T06:47:32.325Z" }, + { url = "https://files.pythonhosted.org/packages/86/c2/dec9722dc3474c164a0b6bcd9a7ed7da542c98af8cabce05374abab35edd/multiprocess-0.70.19-py311-none-any.whl", hash = "sha256:928851ae7973aea4ce0eaf330bbdafb2e01398a91518d5c8818802845564f45c", size = 144457, upload-time = "2026-01-19T06:47:33.711Z" }, + { url = "https://files.pythonhosted.org/packages/71/70/38998b950a97ea279e6bd657575d22d1a2047256caf707d9a10fbce4f065/multiprocess-0.70.19-py312-none-any.whl", hash = "sha256:3a56c0e85dd5025161bac5ce138dcac1e49174c7d8e74596537e729fd5c53c28", size = 150281, upload-time = "2026-01-19T06:47:35.037Z" }, + { url = "https://files.pythonhosted.org/packages/7f/74/d2c27e03cb84251dfe7249b8e82923643c6d48fa4883b9476b025e7dc7eb/multiprocess-0.70.19-py313-none-any.whl", hash = "sha256:8d5eb4ec5017ba2fab4e34a747c6d2c2b6fecfe9e7236e77988db91580ada952", size = 156414, upload-time = "2026-01-19T06:47:35.915Z" }, + { url = "https://files.pythonhosted.org/packages/a0/61/af9115673a5870fd885247e2f1b68c4f1197737da315b520a91c757a861a/multiprocess-0.70.19-py314-none-any.whl", hash = "sha256:e8cc7fbdff15c0613f0a1f1f8744bef961b0a164c0ca29bdff53e9d2d93c5e5f", size = 160318, upload-time = "2026-01-19T06:47:37.497Z" }, + { url = "https://files.pythonhosted.org/packages/7e/82/69e539c4c2027f1e1697e09aaa2449243085a0edf81ae2c6341e84d769b6/multiprocess-0.70.19-py39-none-any.whl", hash = "sha256:0d4b4397ed669d371c81dcd1ef33fd384a44d6c3de1bd0ca7ac06d837720d3c5", size = 133477, upload-time = "2026-01-19T06:47:38.619Z" }, +] + [[package]] name = "multivolumefile" version = "0.2.3" @@ -2669,6 +2931,24 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/22/31/ec5f46fd4c83185b806aa9c736e228cb780f13990a9cf4da0beb70025fcc/multivolumefile-0.2.3-py3-none-any.whl", hash = "sha256:237f4353b60af1703087cf7725755a1f6fcaeeea48421e1896940cd1c920d678", size = 17037, upload-time = "2021-04-29T12:18:38.886Z" }, ] +[[package]] +name = "nest-asyncio" +version = "1.6.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/83/f8/51569ac65d696c8ecbee95938f89d4abf00f47d58d48f6fbabfe8f0baefe/nest_asyncio-1.6.0.tar.gz", hash = "sha256:6f172d5449aca15afd6c646851f4e31e02c598d553a667e38cafa997cfec55fe", size = 7418, upload-time = "2024-01-21T14:25:19.227Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a0/c4/c2971a3ba4c6103a3d10c4b0f24f461ddc027f0f09763220cf35ca1401b3/nest_asyncio-1.6.0-py3-none-any.whl", hash = "sha256:87af6efd6b5e897c81050477ef65c62e2b2f35d51703cae01aff2905b1852e1c", size = 5195, upload-time = "2024-01-21T14:25:17.223Z" }, +] + +[[package]] +name = "networkx" +version = "3.6.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/6a/51/63fe664f3908c97be9d2e4f1158eb633317598cfa6e1fc14af5383f17512/networkx-3.6.1.tar.gz", hash = "sha256:26b7c357accc0c8cde558ad486283728b65b6a95d85ee1cd66bafab4c8168509", size = 2517025, upload-time = "2025-12-08T17:02:39.908Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9e/c9/b2622292ea83fbb4ec318f5b9ab867d0a28ab43c5717bb85b0a5f6b3b0a4/networkx-3.6.1-py3-none-any.whl", hash = "sha256:d47fbf302e7d9cbbb9e2555a0d267983d2aa476bac30e90dfbe5669bd57f3762", size = 2068504, upload-time = "2025-12-08T17:02:38.159Z" }, +] + [[package]] name = "nexus-rpc" version = "1.4.0" @@ -2775,6 +3055,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/ac/ff/05257b7183279b80ecec6333744de23f48f0faeeba46c93e6d13ce835515/oletools-0.60.2-py2.py3-none-any.whl", hash = "sha256:72ad8bd748fd0c4e7b5b4733af770d11543ebb2bf2697455f99f975fcd50cc96", size = 989449, upload-time = "2024-07-02T14:50:29.122Z" }, ] +[[package]] +name = "ollama" +version = "0.6.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "httpx" }, + { name = "pydantic" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/fc/72/5f12423b6b39ca8430fbe56f77fcf4ef60f63067c7c4a2e30e200ed9ec16/ollama-0.6.2.tar.gz", hash = "sha256:936d55daa684f474364c098611c933626f8d6c7d67065c5b7ae0c477b508b07f", size = 53145, upload-time = "2026-04-29T21:21:15.018Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c4/ab/d6722beeb2d10f7a3b9ff49375708904fde18f82b5609a0bc4aeb5996a4d/ollama-0.6.2-py3-none-any.whl", hash = "sha256:3ad7daab28e5a973445c36a73882a3ef698c2ebb00e21e308652741577509f7d", size = 15115, upload-time = "2026-04-29T21:21:13.794Z" }, +] + [[package]] name = "onnxruntime" version = "1.26.0" @@ -3011,6 +3304,36 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/16/21/5a3f1e8913103b703a436a5664238e5b965ec392b555fe68943ea3691e6b/orjson-3.11.9-cp314-cp314-win_arm64.whl", hash = "sha256:eebdbdeef0094e4f5aefa20dcd4eb2368ab5e7a3b4edea27f1e7b2892e009cf9", size = 126687, upload-time = "2026-05-06T15:11:06.602Z" }, ] +[[package]] +name = "ormsgpack" +version = "1.12.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/12/0c/f1761e21486942ab9bb6feaebc610fa074f7c5e496e6962dea5873348077/ormsgpack-1.12.2.tar.gz", hash = "sha256:944a2233640273bee67521795a73cf1e959538e0dfb7ac635505010455e53b33", size = 39031, upload-time = "2026-01-18T20:55:28.023Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/eb/29/bb0eba3288c0449efbb013e9c6f58aea79cf5cb9ee1921f8865f04c1a9d7/ormsgpack-1.12.2-cp313-cp313-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:5ea60cb5f210b1cfbad8c002948d73447508e629ec375acb82910e3efa8ff355", size = 378661, upload-time = "2026-01-18T20:55:57.765Z" }, + { url = "https://files.pythonhosted.org/packages/6e/31/5efa31346affdac489acade2926989e019e8ca98129658a183e3add7af5e/ormsgpack-1.12.2-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f3601f19afdbea273ed70b06495e5794606a8b690a568d6c996a90d7255e51c1", size = 203194, upload-time = "2026-01-18T20:56:08.252Z" }, + { url = "https://files.pythonhosted.org/packages/eb/56/d0087278beef833187e0167f8527235ebe6f6ffc2a143e9de12a98b1ce87/ormsgpack-1.12.2-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:29a9f17a3dac6054c0dce7925e0f4995c727f7c41859adf9b5572180f640d172", size = 210778, upload-time = "2026-01-18T20:55:17.694Z" }, + { url = "https://files.pythonhosted.org/packages/1c/a2/072343e1413d9443e5a252a8eb591c2d5b1bffbe5e7bfc78c069361b92eb/ormsgpack-1.12.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:39c1bd2092880e413902910388be8715f70b9f15f20779d44e673033a6146f2d", size = 212592, upload-time = "2026-01-18T20:55:32.747Z" }, + { url = "https://files.pythonhosted.org/packages/a2/8b/a0da3b98a91d41187a63b02dda14267eefc2a74fcb43cc2701066cf1510e/ormsgpack-1.12.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:50b7249244382209877deedeee838aef1542f3d0fc28b8fe71ca9d7e1896a0d7", size = 387164, upload-time = "2026-01-18T20:55:40.853Z" }, + { url = "https://files.pythonhosted.org/packages/19/bb/6d226bc4cf9fc20d8eb1d976d027a3f7c3491e8f08289a2e76abe96a65f3/ormsgpack-1.12.2-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:5af04800d844451cf102a59c74a841324868d3f1625c296a06cc655c542a6685", size = 482516, upload-time = "2026-01-18T20:55:42.033Z" }, + { url = "https://files.pythonhosted.org/packages/fb/f1/bb2c7223398543dedb3dbf8bb93aaa737b387de61c5feaad6f908841b782/ormsgpack-1.12.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:cec70477d4371cd524534cd16472d8b9cc187e0e3043a8790545a9a9b296c258", size = 425539, upload-time = "2026-01-18T20:55:24.727Z" }, + { url = "https://files.pythonhosted.org/packages/7b/e8/0fb45f57a2ada1fed374f7494c8cd55e2f88ccd0ab0a669aa3468716bf5f/ormsgpack-1.12.2-cp313-cp313-win_amd64.whl", hash = "sha256:21f4276caca5c03a818041d637e4019bc84f9d6ca8baa5ea03e5cc8bf56140e9", size = 117459, upload-time = "2026-01-18T20:55:56.876Z" }, + { url = "https://files.pythonhosted.org/packages/7a/d4/0cfeea1e960d550a131001a7f38a5132c7ae3ebde4c82af1f364ccc5d904/ormsgpack-1.12.2-cp313-cp313-win_arm64.whl", hash = "sha256:baca4b6773d20a82e36d6fd25f341064244f9f86a13dead95dd7d7f996f51709", size = 111577, upload-time = "2026-01-18T20:55:43.605Z" }, + { url = "https://files.pythonhosted.org/packages/94/16/24d18851334be09c25e87f74307c84950f18c324a4d3c0b41dabdbf19c29/ormsgpack-1.12.2-cp314-cp314-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:bc68dd5915f4acf66ff2010ee47c8906dc1cf07399b16f4089f8c71733f6e36c", size = 378717, upload-time = "2026-01-18T20:55:26.164Z" }, + { url = "https://files.pythonhosted.org/packages/b5/a2/88b9b56f83adae8032ac6a6fa7f080c65b3baf9b6b64fd3d37bd202991d4/ormsgpack-1.12.2-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:46d084427b4132553940070ad95107266656cb646ea9da4975f85cb1a6676553", size = 203183, upload-time = "2026-01-18T20:55:18.815Z" }, + { url = "https://files.pythonhosted.org/packages/a9/80/43e4555963bf602e5bdc79cbc8debd8b6d5456c00d2504df9775e74b450b/ormsgpack-1.12.2-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:c010da16235806cf1d7bc4c96bf286bfa91c686853395a299b3ddb49499a3e13", size = 210814, upload-time = "2026-01-18T20:55:33.973Z" }, + { url = "https://files.pythonhosted.org/packages/78/e1/7cfbf28de8bca6efe7e525b329c31277d1b64ce08dcba723971c241a9d60/ormsgpack-1.12.2-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:18867233df592c997154ff942a6503df274b5ac1765215bceba7a231bea2745d", size = 212634, upload-time = "2026-01-18T20:55:28.634Z" }, + { url = "https://files.pythonhosted.org/packages/95/f8/30ae5716e88d792a4e879debee195653c26ddd3964c968594ddef0a3cc7e/ormsgpack-1.12.2-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:b009049086ddc6b8f80c76b3955df1aa22a5fbd7673c525cd63bf91f23122ede", size = 387139, upload-time = "2026-01-18T20:56:02.013Z" }, + { url = "https://files.pythonhosted.org/packages/dc/81/aee5b18a3e3a0e52f718b37ab4b8af6fae0d9d6a65103036a90c2a8ffb5d/ormsgpack-1.12.2-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:1dcc17d92b6390d4f18f937cf0b99054824a7815818012ddca925d6e01c2e49e", size = 482578, upload-time = "2026-01-18T20:55:35.117Z" }, + { url = "https://files.pythonhosted.org/packages/bd/17/71c9ba472d5d45f7546317f467a5fc941929cd68fb32796ca3d13dcbaec2/ormsgpack-1.12.2-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:f04b5e896d510b07c0ad733d7fce2d44b260c5e6c402d272128f8941984e4285", size = 425539, upload-time = "2026-01-18T20:56:04.009Z" }, + { url = "https://files.pythonhosted.org/packages/2e/a6/ac99cd7fe77e822fed5250ff4b86fa66dd4238937dd178d2299f10b69816/ormsgpack-1.12.2-cp314-cp314-win_amd64.whl", hash = "sha256:ae3aba7eed4ca7cb79fd3436eddd29140f17ea254b91604aa1eb19bfcedb990f", size = 117493, upload-time = "2026-01-18T20:56:07.343Z" }, + { url = "https://files.pythonhosted.org/packages/3a/67/339872846a1ae4592535385a1c1f93614138566d7af094200c9c3b45d1e5/ormsgpack-1.12.2-cp314-cp314-win_arm64.whl", hash = "sha256:118576ea6006893aea811b17429bfc561b4778fad393f5f538c84af70b01260c", size = 111579, upload-time = "2026-01-18T20:55:21.161Z" }, + { url = "https://files.pythonhosted.org/packages/49/c2/6feb972dc87285ad381749d3882d8aecbde9f6ecf908dd717d33d66df095/ormsgpack-1.12.2-cp314-cp314t-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:7121b3d355d3858781dc40dafe25a32ff8a8242b9d80c692fd548a4b1f7fd3c8", size = 378721, upload-time = "2026-01-18T20:55:52.12Z" }, + { url = "https://files.pythonhosted.org/packages/a3/9a/900a6b9b413e0f8a471cf07830f9cf65939af039a362204b36bd5b581d8b/ormsgpack-1.12.2-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4ee766d2e78251b7a63daf1cddfac36a73562d3ddef68cacfb41b2af64698033", size = 203170, upload-time = "2026-01-18T20:55:44.469Z" }, + { url = "https://files.pythonhosted.org/packages/87/4c/27a95466354606b256f24fad464d7c97ab62bce6cc529dd4673e1179b8fb/ormsgpack-1.12.2-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:292410a7d23de9b40444636b9b8f1e4e4b814af7f1ef476e44887e52a123f09d", size = 212816, upload-time = "2026-01-18T20:55:23.501Z" }, + { url = "https://files.pythonhosted.org/packages/73/cd/29cee6007bddf7a834e6cd6f536754c0535fcb939d384f0f37a38b1cddb8/ormsgpack-1.12.2-cp314-cp314t-win_amd64.whl", hash = "sha256:837dd316584485b72ef451d08dd3e96c4a11d12e4963aedb40e08f89685d8ec2", size = 117232, upload-time = "2026-01-18T20:55:45.448Z" }, +] + [[package]] name = "overrides" version = "7.7.0" @@ -3476,6 +3799,42 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/f0/9c/762284710ead9076eeecd55fb60509c19cd1f4bea811df5f3603725b44cb/py7zr-1.1.0-py3-none-any.whl", hash = "sha256:5921bc30fb72b5453aafe3b2183664c08ef508cde2655988d5e9bd6078353ef7", size = 71257, upload-time = "2025-12-21T03:27:42.881Z" }, ] +[[package]] +name = "pyarrow" +version = "24.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/91/13/13e1069b351bdc3881266e11147ffccf687505dbb0ea74036237f5d454a5/pyarrow-24.0.0.tar.gz", hash = "sha256:85fe721a14dd823aca09127acbb06c3ca723efbd436c004f16bca601b04dcc83", size = 1180261, upload-time = "2026-04-21T10:51:25.837Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6f/d3/a1abf004482026ddc17f4503db227787fa3cfe41ec5091ff20e4fea55e57/pyarrow-24.0.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:02b001b3ed4723caa44f6cd1af2d5c86aa2cf9971dacc2ffa55b21237713dfba", size = 34976759, upload-time = "2026-04-21T10:48:07.258Z" }, + { url = "https://files.pythonhosted.org/packages/4f/4a/34f0a36d28a2dd32225301b79daad44e243dc1a2bb77d43b60749be255c4/pyarrow-24.0.0-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:04920d6a71aabd08a0417709efce97d45ea8e6fb733d9ca9ecffb13c67839f68", size = 36658471, upload-time = "2026-04-21T10:48:13.347Z" }, + { url = "https://files.pythonhosted.org/packages/1f/78/543b94712ae8bb1a6023bcc1acf1a740fbff8286747c289cd9468fced2a5/pyarrow-24.0.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:a964266397740257f16f7bb2e4f08a0c81454004beab8ff59dd531b73610e9f2", size = 45675981, upload-time = "2026-04-21T10:48:20.201Z" }, + { url = "https://files.pythonhosted.org/packages/84/9f/8fb7c222b100d314137fa40ec050de56cd8c6d957d1cfff685ce72f15b17/pyarrow-24.0.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:6f066b179d68c413374294bc1735f68475457c933258df594443bb9d88ddc2a0", size = 48859172, upload-time = "2026-04-21T10:48:27.541Z" }, + { url = "https://files.pythonhosted.org/packages/a7/d3/1ea72538e6c8b3b475ed78d1049a2c518e655761ea50fe1171fc855fcab7/pyarrow-24.0.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:1183baeb14c5f587b1ec52831e665718ce632caab84b7cd6b85fd44f96114495", size = 49385733, upload-time = "2026-04-21T10:48:34.7Z" }, + { url = "https://files.pythonhosted.org/packages/c3/be/c3d8b06a1ba35f2260f8e1f771abbee7d5e345c0937aab90675706b1690a/pyarrow-24.0.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:806f24b4085453c197a5078218d1ee08783ebbba271badd153d1ae22a3ee804f", size = 51934335, upload-time = "2026-04-21T10:48:42.099Z" }, + { url = "https://files.pythonhosted.org/packages/9c/62/89e07a1e7329d2cde3e3c6994ba0839a24977a2beda8be6005ea3d860b99/pyarrow-24.0.0-cp313-cp313-win_amd64.whl", hash = "sha256:e4505fc6583f7b05ab854934896bcac8253b04ac1171a77dfb73efef92076d91", size = 27271748, upload-time = "2026-04-21T10:49:42.532Z" }, + { url = "https://files.pythonhosted.org/packages/17/1a/cff3a59f80b5b1658549d46611b67163f65e0664431c076ad728bf9d5af4/pyarrow-24.0.0-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:1a4e45017efbf115032e4475ee876d525e0e36c742214fbe405332480ecd6275", size = 35238554, upload-time = "2026-04-21T10:48:48.526Z" }, + { url = "https://files.pythonhosted.org/packages/a8/99/cce0f42a327bfef2c420fb6078a3eb834826e5d6697bf3009fe11d2ad051/pyarrow-24.0.0-cp313-cp313t-macosx_12_0_x86_64.whl", hash = "sha256:7986f1fa71cee060ad00758bcc79d3a93bab8559bf978fab9e53472a2e25a17b", size = 36782301, upload-time = "2026-04-21T10:48:55.181Z" }, + { url = "https://files.pythonhosted.org/packages/2a/66/8e560d5ff6793ca29aca213c53eec0dd482dd46cb93b2819e5aab52e4252/pyarrow-24.0.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:d3e0b61e8efb24ed38898e5cdc5fffa9124be480008d401a1f8071500494ae42", size = 45721929, upload-time = "2026-04-21T10:49:03.676Z" }, + { url = "https://files.pythonhosted.org/packages/27/0c/a26e25505d030716e078d9f16eb74973cbf0b33b672884e9f9da1c83b871/pyarrow-24.0.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:55a3bc1e3df3b5567b7d27ef551b2283f0c68a5e86f1cd56abc569da4f31335b", size = 48825365, upload-time = "2026-04-21T10:49:11.714Z" }, + { url = "https://files.pythonhosted.org/packages/5f/eb/771f9ecb0c65e73fe9dccdd1717901b9594f08c4515d000c7c62df573811/pyarrow-24.0.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:641f795b361874ac9da5294f8f443dfdbee355cf2bd9e3b8d97aaac2306b9b37", size = 49451819, upload-time = "2026-04-21T10:49:21.474Z" }, + { url = "https://files.pythonhosted.org/packages/48/da/61ae89a88732f5a785646f3ec6125dbb640fa98a540eb2b9889caa561403/pyarrow-24.0.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:8adc8e6ce5fccf5dc707046ae4914fd537def529709cc0d285d37a7f9cd442ca", size = 51909252, upload-time = "2026-04-21T10:49:31.164Z" }, + { url = "https://files.pythonhosted.org/packages/cb/1a/8dd5cafab7b66573fa91c03d06d213356ad4edd71813aa75e08ce2b3a844/pyarrow-24.0.0-cp313-cp313t-win_amd64.whl", hash = "sha256:9b18371ad2f44044b81a8d23bc2d8a9b6a6226dca775e8e16cfee640473d6c5d", size = 27388127, upload-time = "2026-04-21T10:49:37.334Z" }, + { url = "https://files.pythonhosted.org/packages/ad/80/d022a34ff05d2cbedd8ccf841fc1f532ecfa9eb5ed1711b56d0e0ea71fc9/pyarrow-24.0.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:1cc9057f0319e26333b357e17f3c2c022f1a83739b48a88b25bfd5fa2dc18838", size = 35007997, upload-time = "2026-04-21T10:49:48.796Z" }, + { url = "https://files.pythonhosted.org/packages/1a/ff/f01485fda6f4e5d441afb8dd5e7681e4db18826c1e271852f5d3957d6a80/pyarrow-24.0.0-cp314-cp314-macosx_12_0_x86_64.whl", hash = "sha256:e6f1278ee4785b6db21229374a1c9e54ec7c549de5d1efc9630b6207de7e170b", size = 36678720, upload-time = "2026-04-21T10:49:55.858Z" }, + { url = "https://files.pythonhosted.org/packages/9e/c2/2d2d5fea814237923f71b36495211f20b43a1576f9a4d6da7e751a64ec6f/pyarrow-24.0.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:adbbedc55506cbdabb830890444fb856bfb0060c46c6f8026c6c2f2cf86ae795", size = 45741852, upload-time = "2026-04-21T10:50:04.624Z" }, + { url = "https://files.pythonhosted.org/packages/8e/3a/28ba9c1c1ebdbb5f1b94dfebb46f207e52e6a554b7fe4132540fde29a3a0/pyarrow-24.0.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:ae8a1145af31d903fa9bb166824d7abe9b4681a000b0159c9fb99c11bc11ad26", size = 48889852, upload-time = "2026-04-21T10:50:12.293Z" }, + { url = "https://files.pythonhosted.org/packages/df/51/4a389acfd31dca009f8fb82d7f510bb4130f2b3a8e18cf00194d0687d8ac/pyarrow-24.0.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:d7027eba1df3b2069e2e8d80f644fa0918b68c46432af3d088ddd390d063ecde", size = 49445207, upload-time = "2026-04-21T10:50:20.677Z" }, + { url = "https://files.pythonhosted.org/packages/19/4b/0bab2b23d2ae901b1b9a03c0efd4b2d070256f8ce3fc43f6e58c167b2081/pyarrow-24.0.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:e56a1ffe9bf7b727432b89104cc0849c21582949dd7bdcb34f17b2001a351a76", size = 51954117, upload-time = "2026-04-21T10:50:29.14Z" }, + { url = "https://files.pythonhosted.org/packages/29/88/f4e9145da0417b3d2c12035a8492b35ff4a3dbc653e614fcfb51d9dedb38/pyarrow-24.0.0-cp314-cp314-win_amd64.whl", hash = "sha256:38be1808cdd068605b787e6ca9119b27eb275a0234e50212c3492331680c3b1e", size = 28001155, upload-time = "2026-04-21T10:51:22.337Z" }, + { url = "https://files.pythonhosted.org/packages/79/4f/46a49a63f43526da895b1a45bbb51d5baf8e4d77159f8528fc3e5490007f/pyarrow-24.0.0-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:418e48ce50a45a6a6c73c454677203a9c75c966cb1e92ca3370959185f197a05", size = 35250387, upload-time = "2026-04-21T10:50:35.552Z" }, + { url = "https://files.pythonhosted.org/packages/a0/da/d5e0cd5ef00796922404806d5f00325cdadc3441ce2c13fe7115f2df9a64/pyarrow-24.0.0-cp314-cp314t-macosx_12_0_x86_64.whl", hash = "sha256:2f16197705a230a78270cdd4ea8a1d57e86b2fdcbc34a1f6aebc72e65c986f9a", size = 36797102, upload-time = "2026-04-21T10:50:42.417Z" }, + { url = "https://files.pythonhosted.org/packages/34/c7/5904145b0a593a05236c882933d439b5720f0a145381179063722fbfc123/pyarrow-24.0.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:fb24ac194bfc5e86839d7dcd52092ee31e5fe6733fe11f5e3b06ef0812b20072", size = 45745118, upload-time = "2026-04-21T10:50:49.324Z" }, + { url = "https://files.pythonhosted.org/packages/13/d3/cca42fe166d1c6e4d5b80e530b7949104d10e17508a90ae202dac205ce2a/pyarrow-24.0.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:9700ebd9a51f5895ce75ff4ac4b3c47a7d4b42bc618be8e713e5d56bacf5f931", size = 48844765, upload-time = "2026-04-21T10:50:55.579Z" }, + { url = "https://files.pythonhosted.org/packages/b0/49/942c3b79878ba928324d1e17c274ed84581db8c0a749b24bcf4cbdf15bd3/pyarrow-24.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:d8ddd2768da81d3ee08cfea9b597f4abb4e8e1dc8ae7e204b608d23a0d3ab699", size = 49471890, upload-time = "2026-04-21T10:51:02.439Z" }, + { url = "https://files.pythonhosted.org/packages/76/97/ff71431000a75d84135a1ace5ca4ba11726a231a8007bbb320a4c54075d5/pyarrow-24.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:61a3d7eaa97a14768b542f3d284dc6400dd2470d9f080708b13cd46b6ae18136", size = 51932250, upload-time = "2026-04-21T10:51:10.576Z" }, + { url = "https://files.pythonhosted.org/packages/51/be/6f79d55816d5c22557cf27533543d5d70dfe692adfbee4b99f2760674f38/pyarrow-24.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:c91d00057f23b8d353039520dc3a6c09d8608164c692e9f59a175a42b2ae0c19", size = 28131282, upload-time = "2026-04-21T10:51:16.815Z" }, +] + [[package]] name = "pyasn1" version = "0.6.3" @@ -4298,6 +4657,36 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d6/10/c437bd2ac41ef30d3019063e6ce537dc111e9214473b337ee88f7fa6359a/qdrant_client-1.18.0-py3-none-any.whl", hash = "sha256:093aa8cf8a420ee3ad2a68b007e1378d7992b2600e0b53c193fc172674f659cd", size = 398126, upload-time = "2026-05-11T14:12:36.998Z" }, ] +[[package]] +name = "ragas" +version = "0.4.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "appdirs" }, + { name = "datasets" }, + { name = "diskcache" }, + { name = "instructor" }, + { name = "langchain" }, + { name = "langchain-community" }, + { name = "langchain-core" }, + { name = "langchain-openai" }, + { name = "nest-asyncio" }, + { name = "networkx" }, + { name = "numpy" }, + { name = "openai" }, + { name = "pillow" }, + { name = "pydantic" }, + { name = "rich" }, + { name = "scikit-network" }, + { name = "tiktoken" }, + { name = "tqdm" }, + { name = "typer" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/d2/bc/3234517692ac0ffae1ec2ec940992e4057844c49ee6c51c07ce385bb98f1/ragas-0.4.3.tar.gz", hash = "sha256:1eb1f61dbc8613ad014fdb8d630cbe9a1caec1ea01664a106993cb756128c001", size = 44029626, upload-time = "2026-01-13T17:48:01.043Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4d/e0/1fecd22c93d3ed66453cbbdefd05528331af4d33b2b76a370d751231912c/ragas-0.4.3-py3-none-any.whl", hash = "sha256:ef1d75f674c294e9a6e7d8e9ad261b6bf4697dad1c9cbd1a756ba7a6b4849a38", size = 466452, upload-time = "2026-01-13T17:47:59.2Z" }, +] + [[package]] name = "red-black-tree-mod" version = "1.22" @@ -4431,15 +4820,15 @@ wheels = [ [[package]] name = "rich" -version = "15.0.0" +version = "14.3.4" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "markdown-it-py" }, { name = "pygments" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/c0/8f/0722ca900cc807c13a6a0c696dacf35430f72e0ec571c4275d2371fca3e9/rich-15.0.0.tar.gz", hash = "sha256:edd07a4824c6b40189fb7ac9bc4c52536e9780fbbfbddf6f1e2502c31b068c36", size = 230680, upload-time = "2026-04-12T08:24:00.75Z" } +sdist = { url = "https://files.pythonhosted.org/packages/e9/67/cae617f1351490c25a4b8ac3b8b63a4dda609295d8222bad12242dfdc629/rich-14.3.4.tar.gz", hash = "sha256:817e02727f2b25b40ef56f5aa2217f400c8489f79ca8f46ea2b70dd5e14558a9", size = 230524, upload-time = "2026-04-11T02:57:45.419Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/82/3b/64d4899d73f91ba49a8c18a8ff3f0ea8f1c1d75481760df8c68ef5235bf5/rich-15.0.0-py3-none-any.whl", hash = "sha256:33bd4ef74232fb73fe9279a257718407f169c09b78a87ad3d296f548e27de0bb", size = 310654, upload-time = "2026-04-12T08:24:02.83Z" }, + { url = "https://files.pythonhosted.org/packages/b3/76/6d163cfac87b632216f71879e6b2cf17163f773ff59c00b5ff4900a80fa3/rich-14.3.4-py3-none-any.whl", hash = "sha256:07e7adb4690f68864777b1450859253bed81a99a31ac321ac1817b2313558952", size = 310480, upload-time = "2026-04-11T02:57:47.484Z" }, ] [[package]] @@ -4571,6 +4960,64 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/87/72/c6c32d2b657fa3dad1de340254e14390b1e334ce38268b7ad51abda3c8c2/s3transfer-0.17.0-py3-none-any.whl", hash = "sha256:ce3801712acf4ad3e89fb9990df97b4972e93f4b3b0004d214be5bce12814c20", size = 86811, upload-time = "2026-04-29T22:07:34.966Z" }, ] +[[package]] +name = "scikit-network" +version = "0.33.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy" }, + { name = "scipy" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/46/6d/28b00fbef9ff7d8ba31861bf16705a1a74a1696fb65aab2a7c584f966bec/scikit_network-0.33.5.tar.gz", hash = "sha256:ae2149d9a280fdc4bbadd5f8a7b17c8af61c054bc3f834792bc61483e6783c12", size = 1784205, upload-time = "2025-11-19T09:45:14.402Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d6/cd/0069244e970d27fa0ab0512394295a106605f00c271e85618182460d2c92/scikit_network-0.33.5-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:aa8b490a777e081dd6f69627a26b50cc4012f27fff683a1e4828819a88a5dcf2", size = 2862564, upload-time = "2025-11-19T09:45:03.93Z" }, + { url = "https://files.pythonhosted.org/packages/51/37/85454864e50a65e528fe3b15eb3b41eb68b0c7f6d5c51c220b3198622ede/scikit_network-0.33.5-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:3615d073ba9ae1ae30dda2de747474cd23c86cededa82b317471ee9f9bebd1b2", size = 2842521, upload-time = "2025-11-19T09:45:06.31Z" }, + { url = "https://files.pythonhosted.org/packages/c4/b9/4023f35e430b51020f17b0f1d8933768cf1ed7cd1623cc089f5543048983/scikit_network-0.33.5-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2408d3f4c81256a3193d536aad4a6ffcfbb05d096abe6a9cc0b6b5e275df876d", size = 7871695, upload-time = "2025-11-19T09:45:08.101Z" }, + { url = "https://files.pythonhosted.org/packages/8e/ec/e50755f7459130ba745c42c37665c5ae9a7c7e357f43400b5b8b966f902e/scikit_network-0.33.5-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:526490a1e0e8e49ad0f4cca193f581d60082a2fa8c9a825eb0b6936050b0d02b", size = 7968762, upload-time = "2025-11-19T09:45:10.347Z" }, + { url = "https://files.pythonhosted.org/packages/45/2a/616974a0adb9d04a791570e9371caaef14c54f8806b04dd59c80a7b60289/scikit_network-0.33.5-cp313-cp313-win_amd64.whl", hash = "sha256:722c15fcede5e07ac008354bbd6ef375e0f5bf1fd52bd40271775997be2fb715", size = 2742482, upload-time = "2025-11-19T09:45:12.843Z" }, +] + +[[package]] +name = "scipy" +version = "1.18.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a7/25/c2700dfaf6442b4effaa91af24ebce5dc9d31bb4a69706313aae70d72cd0/scipy-1.18.0.tar.gz", hash = "sha256:67b2ad2ad54c72ca6d04975a9b2df8c3638c34ddd5b28738e94fc2b57929d378", size = 30774447, upload-time = "2026-06-19T15:01:43.456Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/05/52/9c0136c2de7ae0779b7b366447766cec6d9f0702c56bb8ffeb04c8fd3af4/scipy-1.18.0-cp313-cp313-macosx_10_15_x86_64.whl", hash = "sha256:09143f676d157d9f546d663504ef9c1becb819824f1afc018814176411942446", size = 31036107, upload-time = "2026-06-19T15:00:14.03Z" }, + { url = "https://files.pythonhosted.org/packages/02/73/0291a64843270f4efb86cdcf2ee0f2048631b65ec6b405398b2b4dbf11bf/scipy-1.18.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:5efe260f69417b97ddae455bfb5a95e8359f7f66ad7fa9522a60feb66f169520", size = 28663303, upload-time = "2026-06-19T15:00:16.819Z" }, + { url = "https://files.pythonhosted.org/packages/d3/0f/10ffa0b697a572f4e0d48b92a88895d366422f019f723e7e14a84c050dac/scipy-1.18.0-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:68363b7eaacd8b5dd426df56d782cc156468ac79a127a1b87ca597d6e2e82197", size = 20404960, upload-time = "2026-06-19T15:00:19.635Z" }, + { url = "https://files.pythonhosted.org/packages/7e/d2/e896cea21ba8edd6c81d4c55b1ffcc717e79698dcbebf9641b4cfb4c6622/scipy-1.18.0-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:c5557d8be5da8e41353fcd4d21491fdbab83b062fc579e94dc09a7c8ab4f669b", size = 23034074, upload-time = "2026-06-19T15:00:22.107Z" }, + { url = "https://files.pythonhosted.org/packages/ea/b2/e83ea34279a52c03374477c74006256ec78df65fc877baa4617d6de1d202/scipy-1.18.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0d13bca67c096d89fb95ced0d8921807300fce0275643aef9533cc63a0773468", size = 33942038, upload-time = "2026-06-19T15:00:24.964Z" }, + { url = "https://files.pythonhosted.org/packages/f6/af/e8fe5fb136f51e2b01678b92cb4106d10d8cd68ec147ead2e7cb0ac75398/scipy-1.18.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a46f9273dbd0eb1cefba61c9b8648b4dfe3cbc14a080176f9a73e44b8336dc7f", size = 35266390, upload-time = "2026-06-19T15:00:28.059Z" }, + { url = "https://files.pythonhosted.org/packages/3a/49/2c5cbb907b56695fc67517811d1db234dfd83381a84814ec220aded2794d/scipy-1.18.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:5aba46108853ddfc77906b6557aac839d2b52e900c1d72a1180adaaab58d265f", size = 35551324, upload-time = "2026-06-19T15:00:31.014Z" }, + { url = "https://files.pythonhosted.org/packages/bb/73/eda39f7a2d306ff0ffc574afd13c0bbb6d10a603d9a413998ee269487a80/scipy-1.18.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:b6f758e35f12757b5d95c00bc6de2438e229c2664b7a92e96f205959d9f2dfa4", size = 37404785, upload-time = "2026-06-19T15:00:34.072Z" }, + { url = "https://files.pythonhosted.org/packages/b7/d2/ae881ee28d014f38e0ccbfd974a06a919ba9af34f1f74bf42b5301891d63/scipy-1.18.0-cp313-cp313-win_amd64.whl", hash = "sha256:1afac4a847207c7ff8efd321734a50b06d0280b3b2a2c0fc2f413101747ad7c7", size = 36554943, upload-time = "2026-06-19T15:00:36.903Z" }, + { url = "https://files.pythonhosted.org/packages/70/3a/21154e2d54eb3639c6bf4dbae2e531c68356bfe95990daa30df33b30d556/scipy-1.18.0-cp313-cp313-win_arm64.whl", hash = "sha256:c5dbddf60e58c2312316d097271a8e73d40eaf2eabfa4d95ed7d3695bbf2ce7b", size = 24350911, upload-time = "2026-06-19T15:00:40.062Z" }, + { url = "https://files.pythonhosted.org/packages/78/b5/915a19b3de2f7430062b509653563db1633ddbb6f021b06731521115d4e2/scipy-1.18.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:4c256ee70c0d1a8a2ace807e199ccd4e3f57037433842abb3fb36bc17eaa9578", size = 31036253, upload-time = "2026-06-19T15:00:43.216Z" }, + { url = "https://files.pythonhosted.org/packages/d7/88/b72def7262e150d16be13fca37a96481138d624e700340bc3362a7588929/scipy-1.18.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:2ef3abc54a4ffc53765374b0d5728532dfdd2585ed23f6b11c206a1f0b1b9af8", size = 28673758, upload-time = "2026-06-19T15:00:46.663Z" }, + { url = "https://files.pythonhosted.org/packages/91/02/2e636a61a525632c373cf6a9c24442a3ffb79e364d38e98b32042964ac32/scipy-1.18.0-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:f2a6af57bd9e4a75d70e4117e78a1bbee84f79ae3fbb6d0111005d6ebcc4cb8d", size = 20415514, upload-time = "2026-06-19T15:00:49.399Z" }, + { url = "https://files.pythonhosted.org/packages/c9/b6/2135974442f6aba159d9d39d774a1c8cb19947016725d69fecc685df45bf/scipy-1.18.0-cp314-cp314-macosx_14_0_x86_64.whl", hash = "sha256:3f1ac564d3bf6c03d861d2cd87a1bea0da2887136f7fb1bf519c05a8971452d6", size = 23034398, upload-time = "2026-06-19T15:00:51.941Z" }, + { url = "https://files.pythonhosted.org/packages/f6/e6/ba89ec5abf6ee9257c0d1ec985573f3ae32742c24bc03e016388a40b1b15/scipy-1.18.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:40395a5fcd1abee49a5c7aaa98c29db393eedc835138560a588c47ec16156690", size = 33998032, upload-time = "2026-06-19T15:00:54.838Z" }, + { url = "https://files.pythonhosted.org/packages/7f/c4/bc41eb19b0fd0db868f4132920879019318d80cc522ad8f2bca4611af808/scipy-1.18.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8ca01e8ae69f1b18e9a58d91afead31be3cef0dd905a10249dac559ee15460a0", size = 35283333, upload-time = "2026-06-19T15:00:58.152Z" }, + { url = "https://files.pythonhosted.org/packages/53/a4/cbdeef6eb3830a8462a9d4ada814de5fc984345cc9ecf17cbec51a036f1e/scipy-1.18.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7a7f3b01647384dbc3a711e8c6778e0aabbe93959249fef5c7393396bcac0867", size = 35610216, upload-time = "2026-06-19T15:01:01.155Z" }, + { url = "https://files.pythonhosted.org/packages/80/4d/b2b82502b65f661d1b789c1665dcdf315d5f12194e06fc0b37946294ebae/scipy-1.18.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:6aa94e78ec192a30063a5e72e561c28af769dc311190b24fe91774eff1969709", size = 37418960, upload-time = "2026-06-19T15:01:04.155Z" }, + { url = "https://files.pythonhosted.org/packages/93/3e/902d836831474b0ab5a37d16404f7bc5fafd9efba632890e271ba952635f/scipy-1.18.0-cp314-cp314-win_amd64.whl", hash = "sha256:2d8bbdc6c817f5b4006a54d799d4f5bab6f910193cbb9a1ff310833d4d270f61", size = 37288845, upload-time = "2026-06-19T15:01:07.822Z" }, + { url = "https://files.pythonhosted.org/packages/b6/43/8d73b337a3bdb14daa0314f0434210747c02d79d729ce1777574a817dcf6/scipy-1.18.0-cp314-cp314-win_arm64.whl", hash = "sha256:18e9575f1569b2c54174e6159d32942e03731177f63dce7975f0a0c88d102f5b", size = 24988971, upload-time = "2026-06-19T15:01:11.076Z" }, + { url = "https://files.pythonhosted.org/packages/b4/b4/f11918b0508a2787031a0499a03fbe3546f3bb5ca05d01038c45b278c09a/scipy-1.18.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:f351e0dd702687d12a402b867a1b4146a256923e1c38317cbc472f6372b94707", size = 31399325, upload-time = "2026-06-19T15:01:13.723Z" }, + { url = "https://files.pythonhosted.org/packages/7b/d1/1f287b57c0ff0ee5185dff3946d92c8017d39b0e431f0ae79a3ff1859512/scipy-1.18.0-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:7c7a51b33ce387193c97f228320cf8e87361daa1bba750638677729598b3e677", size = 29092110, upload-time = "2026-06-19T15:01:16.908Z" }, + { url = "https://files.pythonhosted.org/packages/ff/1a/7b74eb6c392fdcb27d414c0e7558a6d0231eb3b6d73571f479bb81ea8794/scipy-1.18.0-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:84031d7b052a54fae2f8632e0ec802073d385476eb9a63079bce6e23ef9283d4", size = 20833811, upload-time = "2026-06-19T15:01:20.488Z" }, + { url = "https://files.pythonhosted.org/packages/7c/ad/f3941716320a7b9cb4d68734a903b45fe16eff5fb7da7e16f2e619304979/scipy-1.18.0-cp314-cp314t-macosx_14_0_x86_64.whl", hash = "sha256:56abf29a7c067dde59be8b9a22d606a4ea1b2f2a4b756d9d903c62818f5dacce", size = 23396644, upload-time = "2026-06-19T15:01:23.364Z" }, + { url = "https://files.pythonhosted.org/packages/22/22/1446b62ffe07f9719b7d9b1b6a4e05a772833ae8f441fe4c22c34c9b250f/scipy-1.18.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1ad44305cfa24b1ba5803cbbebf033590ccbac1aa5d612d727b785325ab408b0", size = 34079318, upload-time = "2026-06-19T15:01:26.002Z" }, + { url = "https://files.pythonhosted.org/packages/56/3b/b87da667098bb470fa30c7011b0ba351ee976dd395c78798c66e941665a3/scipy-1.18.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:945c1761b93f38d7f99ae81ae80c63e621471608c7eeead563f6df025585cd58", size = 35324320, upload-time = "2026-06-19T15:01:28.881Z" }, + { url = "https://files.pythonhosted.org/packages/f8/a1/c7932f91909759b0267f75fdea34e91309f96b895757534b76a90b6b4344/scipy-1.18.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:1a4441f15d620578772a49e5ab48c0ee1f7a0220e387110283062729136b2553", size = 35699541, upload-time = "2026-06-19T15:01:31.968Z" }, + { url = "https://files.pythonhosted.org/packages/f7/86/5185061a1fcc41d18c5dc2463969b3a3964b31d9ac67b2fb05d4c7ff7670/scipy-1.18.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:9aac6192fac56bf2ca534389d24623f07b39ff83317d58287285e7fbd622ff76", size = 37472480, upload-time = "2026-06-19T15:01:35.136Z" }, + { url = "https://files.pythonhosted.org/packages/31/8e/f04c68e39919a010d34f2ee1367fd705b0a25a02f609d755f0bfbc0a15fc/scipy-1.18.0-cp314-cp314t-win_amd64.whl", hash = "sha256:e40baea28ae7f5475c779741e2d90b1247c78531207b49c7030e698ff81cee3f", size = 37365390, upload-time = "2026-06-19T15:01:38.091Z" }, + { url = "https://files.pythonhosted.org/packages/d5/19/969dc072906c84dd0a3b05dcf57ea750936087d7873549e408b35cfc3f97/scipy-1.18.0-cp314-cp314t-win_arm64.whl", hash = "sha256:368e0a705903c466aa5f08eefb39e6b1b6b2d659e7352a31fd9e2438365be0f8", size = 25279661, upload-time = "2026-06-19T15:01:40.817Z" }, +] + [[package]] name = "secretstorage" version = "3.5.0" @@ -5205,38 +5652,22 @@ wheels = [ [[package]] name = "websockets" -version = "16.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/04/24/4b2031d72e840ce4c1ccb255f693b15c334757fc50023e4db9537080b8c4/websockets-16.0.tar.gz", hash = "sha256:5f6261a5e56e8d5c42a4497b364ea24d94d9563e8fbd44e78ac40879c60179b5", size = 179346, upload-time = "2026-01-10T09:23:47.181Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/cc/9c/baa8456050d1c1b08dd0ec7346026668cbc6f145ab4e314d707bb845bf0d/websockets-16.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:878b336ac47938b474c8f982ac2f7266a540adc3fa4ad74ae96fea9823a02cc9", size = 177364, upload-time = "2026-01-10T09:22:59.333Z" }, - { url = "https://files.pythonhosted.org/packages/7e/0c/8811fc53e9bcff68fe7de2bcbe75116a8d959ac699a3200f4847a8925210/websockets-16.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:52a0fec0e6c8d9a784c2c78276a48a2bdf099e4ccc2a4cad53b27718dbfd0230", size = 175039, upload-time = "2026-01-10T09:23:01.171Z" }, - { url = "https://files.pythonhosted.org/packages/aa/82/39a5f910cb99ec0b59e482971238c845af9220d3ab9fa76dd9162cda9d62/websockets-16.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:e6578ed5b6981005df1860a56e3617f14a6c307e6a71b4fff8c48fdc50f3ed2c", size = 175323, upload-time = "2026-01-10T09:23:02.341Z" }, - { url = "https://files.pythonhosted.org/packages/bd/28/0a25ee5342eb5d5f297d992a77e56892ecb65e7854c7898fb7d35e9b33bd/websockets-16.0-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:95724e638f0f9c350bb1c2b0a7ad0e83d9cc0c9259f3ea94e40d7b02a2179ae5", size = 184975, upload-time = "2026-01-10T09:23:03.756Z" }, - { url = "https://files.pythonhosted.org/packages/f9/66/27ea52741752f5107c2e41fda05e8395a682a1e11c4e592a809a90c6a506/websockets-16.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c0204dc62a89dc9d50d682412c10b3542d748260d743500a85c13cd1ee4bde82", size = 186203, upload-time = "2026-01-10T09:23:05.01Z" }, - { url = "https://files.pythonhosted.org/packages/37/e5/8e32857371406a757816a2b471939d51c463509be73fa538216ea52b792a/websockets-16.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:52ac480f44d32970d66763115edea932f1c5b1312de36df06d6b219f6741eed8", size = 185653, upload-time = "2026-01-10T09:23:06.301Z" }, - { url = "https://files.pythonhosted.org/packages/9b/67/f926bac29882894669368dc73f4da900fcdf47955d0a0185d60103df5737/websockets-16.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6e5a82b677f8f6f59e8dfc34ec06ca6b5b48bc4fcda346acd093694cc2c24d8f", size = 184920, upload-time = "2026-01-10T09:23:07.492Z" }, - { url = "https://files.pythonhosted.org/packages/3c/a1/3d6ccdcd125b0a42a311bcd15a7f705d688f73b2a22d8cf1c0875d35d34a/websockets-16.0-cp313-cp313-win32.whl", hash = "sha256:abf050a199613f64c886ea10f38b47770a65154dc37181bfaff70c160f45315a", size = 178255, upload-time = "2026-01-10T09:23:09.245Z" }, - { url = "https://files.pythonhosted.org/packages/6b/ae/90366304d7c2ce80f9b826096a9e9048b4bb760e44d3b873bb272cba696b/websockets-16.0-cp313-cp313-win_amd64.whl", hash = "sha256:3425ac5cf448801335d6fdc7ae1eb22072055417a96cc6b31b3861f455fbc156", size = 178689, upload-time = "2026-01-10T09:23:10.483Z" }, - { url = "https://files.pythonhosted.org/packages/f3/1d/e88022630271f5bd349ed82417136281931e558d628dd52c4d8621b4a0b2/websockets-16.0-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:8cc451a50f2aee53042ac52d2d053d08bf89bcb31ae799cb4487587661c038a0", size = 177406, upload-time = "2026-01-10T09:23:12.178Z" }, - { url = "https://files.pythonhosted.org/packages/f2/78/e63be1bf0724eeb4616efb1ae1c9044f7c3953b7957799abb5915bffd38e/websockets-16.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:daa3b6ff70a9241cf6c7fc9e949d41232d9d7d26fd3522b1ad2b4d62487e9904", size = 175085, upload-time = "2026-01-10T09:23:13.511Z" }, - { url = "https://files.pythonhosted.org/packages/bb/f4/d3c9220d818ee955ae390cf319a7c7a467beceb24f05ee7aaaa2414345ba/websockets-16.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:fd3cb4adb94a2a6e2b7c0d8d05cb94e6f1c81a0cf9dc2694fb65c7e8d94c42e4", size = 175328, upload-time = "2026-01-10T09:23:14.727Z" }, - { url = "https://files.pythonhosted.org/packages/63/bc/d3e208028de777087e6fb2b122051a6ff7bbcca0d6df9d9c2bf1dd869ae9/websockets-16.0-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:781caf5e8eee67f663126490c2f96f40906594cb86b408a703630f95550a8c3e", size = 185044, upload-time = "2026-01-10T09:23:15.939Z" }, - { url = "https://files.pythonhosted.org/packages/ad/6e/9a0927ac24bd33a0a9af834d89e0abc7cfd8e13bed17a86407a66773cc0e/websockets-16.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:caab51a72c51973ca21fa8a18bd8165e1a0183f1ac7066a182ff27107b71e1a4", size = 186279, upload-time = "2026-01-10T09:23:17.148Z" }, - { url = "https://files.pythonhosted.org/packages/b9/ca/bf1c68440d7a868180e11be653c85959502efd3a709323230314fda6e0b3/websockets-16.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:19c4dc84098e523fd63711e563077d39e90ec6702aff4b5d9e344a60cb3c0cb1", size = 185711, upload-time = "2026-01-10T09:23:18.372Z" }, - { url = "https://files.pythonhosted.org/packages/c4/f8/fdc34643a989561f217bb477cbc47a3a07212cbda91c0e4389c43c296ebf/websockets-16.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:a5e18a238a2b2249c9a9235466b90e96ae4795672598a58772dd806edc7ac6d3", size = 184982, upload-time = "2026-01-10T09:23:19.652Z" }, - { url = "https://files.pythonhosted.org/packages/dd/d1/574fa27e233764dbac9c52730d63fcf2823b16f0856b3329fc6268d6ae4f/websockets-16.0-cp314-cp314-win32.whl", hash = "sha256:a069d734c4a043182729edd3e9f247c3b2a4035415a9172fd0f1b71658a320a8", size = 177915, upload-time = "2026-01-10T09:23:21.458Z" }, - { url = "https://files.pythonhosted.org/packages/8a/f1/ae6b937bf3126b5134ce1f482365fde31a357c784ac51852978768b5eff4/websockets-16.0-cp314-cp314-win_amd64.whl", hash = "sha256:c0ee0e63f23914732c6d7e0cce24915c48f3f1512ec1d079ed01fc629dab269d", size = 178381, upload-time = "2026-01-10T09:23:22.715Z" }, - { url = "https://files.pythonhosted.org/packages/06/9b/f791d1db48403e1f0a27577a6beb37afae94254a8c6f08be4a23e4930bc0/websockets-16.0-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:a35539cacc3febb22b8f4d4a99cc79b104226a756aa7400adc722e83b0d03244", size = 177737, upload-time = "2026-01-10T09:23:24.523Z" }, - { url = "https://files.pythonhosted.org/packages/bd/40/53ad02341fa33b3ce489023f635367a4ac98b73570102ad2cdd770dacc9a/websockets-16.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:b784ca5de850f4ce93ec85d3269d24d4c82f22b7212023c974c401d4980ebc5e", size = 175268, upload-time = "2026-01-10T09:23:25.781Z" }, - { url = "https://files.pythonhosted.org/packages/74/9b/6158d4e459b984f949dcbbb0c5d270154c7618e11c01029b9bbd1bb4c4f9/websockets-16.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:569d01a4e7fba956c5ae4fc988f0d4e187900f5497ce46339c996dbf24f17641", size = 175486, upload-time = "2026-01-10T09:23:27.033Z" }, - { url = "https://files.pythonhosted.org/packages/e5/2d/7583b30208b639c8090206f95073646c2c9ffd66f44df967981a64f849ad/websockets-16.0-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:50f23cdd8343b984957e4077839841146f67a3d31ab0d00e6b824e74c5b2f6e8", size = 185331, upload-time = "2026-01-10T09:23:28.259Z" }, - { url = "https://files.pythonhosted.org/packages/45/b0/cce3784eb519b7b5ad680d14b9673a31ab8dcb7aad8b64d81709d2430aa8/websockets-16.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:152284a83a00c59b759697b7f9e9cddf4e3c7861dd0d964b472b70f78f89e80e", size = 186501, upload-time = "2026-01-10T09:23:29.449Z" }, - { url = "https://files.pythonhosted.org/packages/19/60/b8ebe4c7e89fb5f6cdf080623c9d92789a53636950f7abacfc33fe2b3135/websockets-16.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:bc59589ab64b0022385f429b94697348a6a234e8ce22544e3681b2e9331b5944", size = 186062, upload-time = "2026-01-10T09:23:31.368Z" }, - { url = "https://files.pythonhosted.org/packages/88/a8/a080593f89b0138b6cba1b28f8df5673b5506f72879322288b031337c0b8/websockets-16.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:32da954ffa2814258030e5a57bc73a3635463238e797c7375dc8091327434206", size = 185356, upload-time = "2026-01-10T09:23:32.627Z" }, - { url = "https://files.pythonhosted.org/packages/c2/b6/b9afed2afadddaf5ebb2afa801abf4b0868f42f8539bfe4b071b5266c9fe/websockets-16.0-cp314-cp314t-win32.whl", hash = "sha256:5a4b4cc550cb665dd8a47f868c8d04c8230f857363ad3c9caf7a0c3bf8c61ca6", size = 178085, upload-time = "2026-01-10T09:23:33.816Z" }, - { url = "https://files.pythonhosted.org/packages/9f/3e/28135a24e384493fa804216b79a6a6759a38cc4ff59118787b9fb693df93/websockets-16.0-cp314-cp314t-win_amd64.whl", hash = "sha256:b14dc141ed6d2dde437cddb216004bcac6a1df0935d79656387bd41632ba0bbd", size = 178531, upload-time = "2026-01-10T09:23:35.016Z" }, - { url = "https://files.pythonhosted.org/packages/6f/28/258ebab549c2bf3e64d2b0217b973467394a9cea8c42f70418ca2c5d0d2e/websockets-16.0-py3-none-any.whl", hash = "sha256:1637db62fad1dc833276dded54215f2c7fa46912301a24bd94d45d46a011ceec", size = 171598, upload-time = "2026-01-10T09:23:45.395Z" }, +version = "15.0.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/21/e6/26d09fab466b7ca9c7737474c52be4f76a40301b08362eb2dbc19dcc16c1/websockets-15.0.1.tar.gz", hash = "sha256:82544de02076bafba038ce055ee6412d68da13ab47f0c60cab827346de828dee", size = 177016, upload-time = "2025-03-05T20:03:41.606Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cb/9f/51f0cf64471a9d2b4d0fc6c534f323b664e7095640c34562f5182e5a7195/websockets-15.0.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:ee443ef070bb3b6ed74514f5efaa37a252af57c90eb33b956d35c8e9c10a1931", size = 175440, upload-time = "2025-03-05T20:02:36.695Z" }, + { url = "https://files.pythonhosted.org/packages/8a/05/aa116ec9943c718905997412c5989f7ed671bc0188ee2ba89520e8765d7b/websockets-15.0.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:5a939de6b7b4e18ca683218320fc67ea886038265fd1ed30173f5ce3f8e85675", size = 173098, upload-time = "2025-03-05T20:02:37.985Z" }, + { url = "https://files.pythonhosted.org/packages/ff/0b/33cef55ff24f2d92924923c99926dcce78e7bd922d649467f0eda8368923/websockets-15.0.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:746ee8dba912cd6fc889a8147168991d50ed70447bf18bcda7039f7d2e3d9151", size = 173329, upload-time = "2025-03-05T20:02:39.298Z" }, + { url = "https://files.pythonhosted.org/packages/31/1d/063b25dcc01faa8fada1469bdf769de3768b7044eac9d41f734fd7b6ad6d/websockets-15.0.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:595b6c3969023ecf9041b2936ac3827e4623bfa3ccf007575f04c5a6aa318c22", size = 183111, upload-time = "2025-03-05T20:02:40.595Z" }, + { url = "https://files.pythonhosted.org/packages/93/53/9a87ee494a51bf63e4ec9241c1ccc4f7c2f45fff85d5bde2ff74fcb68b9e/websockets-15.0.1-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:3c714d2fc58b5ca3e285461a4cc0c9a66bd0e24c5da9911e30158286c9b5be7f", size = 182054, upload-time = "2025-03-05T20:02:41.926Z" }, + { url = "https://files.pythonhosted.org/packages/ff/b2/83a6ddf56cdcbad4e3d841fcc55d6ba7d19aeb89c50f24dd7e859ec0805f/websockets-15.0.1-cp313-cp313-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0f3c1e2ab208db911594ae5b4f79addeb3501604a165019dd221c0bdcabe4db8", size = 182496, upload-time = "2025-03-05T20:02:43.304Z" }, + { url = "https://files.pythonhosted.org/packages/98/41/e7038944ed0abf34c45aa4635ba28136f06052e08fc2168520bb8b25149f/websockets-15.0.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:229cf1d3ca6c1804400b0a9790dc66528e08a6a1feec0d5040e8b9eb14422375", size = 182829, upload-time = "2025-03-05T20:02:48.812Z" }, + { url = "https://files.pythonhosted.org/packages/e0/17/de15b6158680c7623c6ef0db361da965ab25d813ae54fcfeae2e5b9ef910/websockets-15.0.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:756c56e867a90fb00177d530dca4b097dd753cde348448a1012ed6c5131f8b7d", size = 182217, upload-time = "2025-03-05T20:02:50.14Z" }, + { url = "https://files.pythonhosted.org/packages/33/2b/1f168cb6041853eef0362fb9554c3824367c5560cbdaad89ac40f8c2edfc/websockets-15.0.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:558d023b3df0bffe50a04e710bc87742de35060580a293c2a984299ed83bc4e4", size = 182195, upload-time = "2025-03-05T20:02:51.561Z" }, + { url = "https://files.pythonhosted.org/packages/86/eb/20b6cdf273913d0ad05a6a14aed4b9a85591c18a987a3d47f20fa13dcc47/websockets-15.0.1-cp313-cp313-win32.whl", hash = "sha256:ba9e56e8ceeeedb2e080147ba85ffcd5cd0711b89576b83784d8605a7df455fa", size = 176393, upload-time = "2025-03-05T20:02:53.814Z" }, + { url = "https://files.pythonhosted.org/packages/1b/6c/c65773d6cab416a64d191d6ee8a8b1c68a09970ea6909d16965d26bfed1e/websockets-15.0.1-cp313-cp313-win_amd64.whl", hash = "sha256:e09473f095a819042ecb2ab9465aee615bd9c2028e4ef7d933600a8401c79561", size = 176837, upload-time = "2025-03-05T20:02:55.237Z" }, + { url = "https://files.pythonhosted.org/packages/fa/a8/5b41e0da817d64113292ab1f8247140aac61cbf6cfd085d6a0fa77f4984f/websockets-15.0.1-py3-none-any.whl", hash = "sha256:f7a866fbc1e97b5c617ee4116daaa09b722101d4a3c170c787450ba409f9736f", size = 169743, upload-time = "2025-03-05T20:03:39.41Z" }, ] [[package]]