From 5426a8f80437364cf9b0c33cd1e52040711fafa8 Mon Sep 17 00:00:00 2001 From: Embedded Risk Analytics Date: Thu, 3 Sep 2026 08:32:55 -0700 Subject: [PATCH 1/4] docs: add Fathom committed-state read to Observability --- docs/edge/en/observability/fathom.mdx | 95 +++++++++++++++++++++++++++ 1 file changed, 95 insertions(+) create mode 100644 docs/edge/en/observability/fathom.mdx diff --git a/docs/edge/en/observability/fathom.mdx b/docs/edge/en/observability/fathom.mdx new file mode 100644 index 0000000000..5672bf60fd --- /dev/null +++ b/docs/edge/en/observability/fathom.mdx @@ -0,0 +1,95 @@ +--- +title: Fathom Committed-State Read +description: Check whether a crew's later actions contradict the records it already committed, from the events the crew already emits. +icon: check-double +mode: "wide" +--- + +# Fathom Committed-State Read + +## Overview + +[Fathom](https://github.com/ERA-Fathom/fathom) reads the tool calls and task completions a crew emits, folds the successful ones into the crew's committed state, and names the step where a later action contradicts an earlier commitment. That includes a record written against a key the crew already renamed, a task reported complete while its record was never written, an entity added to a collection that already holds it, and a value the crew replaced and then wrote again. The read is deterministic and needs no model access. It runs after the crew finishes, on a file you already have, and touches nothing in the crew's runtime path. + +## What it catches + +| Finding | The crew... | +|---|---| +| `stale_reference` | acts on a record it already removed or renamed away | +| `superseded_value` | writes a value it already replaced | +| `residual` | ends the run with a record still carrying a value it replaced elsewhere | +| `duplicate_commit` | adds an entity a collection already holds | +| `post_commit_mutation` | changes a record after committing it | + +A crew whose committed state stays consistent returns `coherent`, and the read prints nothing else. + +## Installation + +```shell +uv add fathom-read +``` + +The package ships with a rate-limited demo key. For your own key, write to contact@embeddedriskanalytics.com and set `FATHOM_API_KEY`. + +## Capture the crew's events + +`FathomListener` records `tool_usage_finished`, `tool_usage_error`, and `task_completed` from the CrewAI event bus and writes them to a file when you close it. Keep a reference to the listener while the crew runs. + +```python Code +from crewai import Agent, Task, Crew +from fathom_read.capture.crewai import FathomListener + +listener = FathomListener("events.json") + +crew = Crew(agents=[...], tasks=[...]) +result = crew.kickoff() + +listener.close() +``` + +## Run the read + +```shell +fathom read events.json --format crewai +``` + +If the crew's job involves a rename or a migration, tell the read which token replaced which, so it can also report records that still carry the old value when the run ends. + +```shell +fathom read events.json --format crewai --supersede guest_id=customer_id +``` + +`fathom read` exits 0 when the committed state is coherent and 2 when it finds a contradiction, so it drops into a test suite or a CI step as it is. Add `--json` for a machine-readable verdict, or `--ops` to see the action stream before anything is sent. + +## Map your own tools + +The read knows common record-keeping tool names (`write_record`, `update_record`, `set_value`, `rename_key`, and others). For tools with your own names, map them once. + +```json tools.json +{"save_decision": {"op": "set", "kind": "decision", "key": "topic", "value": "text"}, + "book_seat": {"op": "add", "kind": "flight", "key": "seats", "value": "seat"}, + "confirm_booking": {"op": "commit", "kind": "flight", "key": "booking"}} +``` + +```shell +fathom read events.json --format crewai --map tools.json +``` + +## Example verdict + +The [coherence census](https://github.com/ERA-Fathom/coherence-census) ships a minimal CrewAI event log in which a crew renames a key across three records, writes the third one on the old key, and reports the task complete. The read returns the following. + +```text +$ fathom read traces/crewai_events.json --format crewai --supersede guest_id=customer_id +ops read: 4 rejected (no-ops): 0 live facts: 4 +committed state: 1 finding + [residual] end of run: record 'r2' still carries 'guest_id' when the run ends; the agent replaced it with 'customer_id' elsewhere. +``` + +The full study, with sequential and hierarchical crews across 5, 12, and 20 coupled sub-records on two models, is at [Reading a crew's committed state from its own event stream](https://embeddedriskanalytics.com/research-reading-a-crews-committed-state-from-its-own-event-stream.html). + +## Resources + +- [fathom-read on GitHub](https://github.com/ERA-Fathom/fathom) +- [Coherence census](https://github.com/ERA-Fathom/coherence-census), every framework the read has run on, with a trace per row +- [Embedded Risk Analytics research](https://embeddedriskanalytics.com/research.html) From 80163426d4880e95fc79fb1657b7ef12f0ff2fb1 Mon Sep 17 00:00:00 2001 From: Embedded Risk Analytics Date: Thu, 3 Sep 2026 08:38:12 -0700 Subject: [PATCH 2/4] docs: add Fathom card to Evaluation & Quality Assurance --- docs/edge/en/observability/overview.mdx | 64 +++++++++++++------------ 1 file changed, 34 insertions(+), 30 deletions(-) diff --git a/docs/edge/en/observability/overview.mdx b/docs/edge/en/observability/overview.mdx index 171a922f28..e1c0fe51fc 100644 --- a/docs/edge/en/observability/overview.mdx +++ b/docs/edge/en/observability/overview.mdx @@ -23,49 +23,53 @@ Observability is crucial for understanding how your CrewAI agents perform, ident - - End-to-end tracing for CrewAI workflows with automatic agent interaction capture. - + +End-to-end tracing for CrewAI workflows with automatic agent interaction capture. + - - OpenTelemetry-native monitoring with cost tracking and performance analytics. - + +OpenTelemetry-native monitoring with cost tracking and performance analytics. + - - Machine learning lifecycle management with tracing and evaluation capabilities. - + +Machine learning lifecycle management with tracing and evaluation capabilities. + - - LLM engineering platform with detailed tracing and analytics. - + +LLM engineering platform with detailed tracing and analytics. + - - Open-source observability for LLMs and agent frameworks. - + +Open-source observability for LLMs and agent frameworks. + - - AI observability platform for monitoring and troubleshooting. - + +AI observability platform for monitoring and troubleshooting. + - - AI gateway with comprehensive monitoring and reliability features. - + +AI gateway with comprehensive monitoring and reliability features. + - - Debug, evaluate, and monitor LLM applications with comprehensive tracing. - + +Debug, evaluate, and monitor LLM applications with comprehensive tracing. + - - Weights & Biases platform for tracking and evaluating AI applications. - + +Weights & Biases platform for tracking and evaluating AI applications. + ### Evaluation & Quality Assurance - - Comprehensive evaluation platform for LLM outputs and agent behaviors. - + +Comprehensive evaluation platform for LLM outputs and agent behaviors. + + + +Deterministic committed-state read that names the step where a crew contradicts a record it already committed. + ## Key Observability Metrics From 4e416c4f43668cafe7f57718180707509d326d9c Mon Sep 17 00:00:00 2001 From: Embedded Risk Analytics Date: Thu, 3 Sep 2026 08:45:48 -0700 Subject: [PATCH 3/4] docs: preserve card indentation, add Fathom card --- docs/edge/en/observability/overview.mdx | 66 ++++++++++++------------- 1 file changed, 33 insertions(+), 33 deletions(-) diff --git a/docs/edge/en/observability/overview.mdx b/docs/edge/en/observability/overview.mdx index e1c0fe51fc..97c888190a 100644 --- a/docs/edge/en/observability/overview.mdx +++ b/docs/edge/en/observability/overview.mdx @@ -23,53 +23,53 @@ Observability is crucial for understanding how your CrewAI agents perform, ident - -End-to-end tracing for CrewAI workflows with automatic agent interaction capture. - + + End-to-end tracing for CrewAI workflows with automatic agent interaction capture. + - -OpenTelemetry-native monitoring with cost tracking and performance analytics. - + + OpenTelemetry-native monitoring with cost tracking and performance analytics. + - -Machine learning lifecycle management with tracing and evaluation capabilities. - + + Machine learning lifecycle management with tracing and evaluation capabilities. + - -LLM engineering platform with detailed tracing and analytics. - + + LLM engineering platform with detailed tracing and analytics. + - -Open-source observability for LLMs and agent frameworks. - + + Open-source observability for LLMs and agent frameworks. + - -AI observability platform for monitoring and troubleshooting. - + + AI observability platform for monitoring and troubleshooting. + - -AI gateway with comprehensive monitoring and reliability features. - + + AI gateway with comprehensive monitoring and reliability features. + - -Debug, evaluate, and monitor LLM applications with comprehensive tracing. - + + Debug, evaluate, and monitor LLM applications with comprehensive tracing. + - -Weights & Biases platform for tracking and evaluating AI applications. - + + Weights & Biases platform for tracking and evaluating AI applications. + ### Evaluation & Quality Assurance - -Comprehensive evaluation platform for LLM outputs and agent behaviors. - + + Comprehensive evaluation platform for LLM outputs and agent behaviors. + - -Deterministic committed-state read that names the step where a crew contradicts a record it already committed. - + + Deterministic committed-state read that names the step where a crew contradicts a record it already committed. + ## Key Observability Metrics From 62964664b214846fe808f88590e3f72b9dc9b13f Mon Sep 17 00:00:00 2001 From: Embedded Risk Analytics Date: Thu, 3 Sep 2026 15:50:48 -0700 Subject: [PATCH 4/4] docs: disclose hosted data transmission and correct coherent-run output --- docs/edge/en/observability/fathom.mdx | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/edge/en/observability/fathom.mdx b/docs/edge/en/observability/fathom.mdx index 5672bf60fd..ce7c6c923a 100644 --- a/docs/edge/en/observability/fathom.mdx +++ b/docs/edge/en/observability/fathom.mdx @@ -21,7 +21,7 @@ mode: "wide" | `duplicate_commit` | adds an entity a collection already holds | | `post_commit_mutation` | changes a record after committing it | -A crew whose committed state stays consistent returns `coherent`, and the read prints nothing else. +A crew whose committed state stays consistent returns `coherent`. The read still prints the operation summary and then reports the committed state as coherent. ## Installation @@ -31,6 +31,10 @@ uv add fathom-read The package ships with a rate-limited demo key. For your own key, write to contact@embeddedriskanalytics.com and set `FATHOM_API_KEY`. +## Data handling + +`fathom read` sends the operation stream to the Fathom hosted read. That stream is derived from the crew's tool calls, so it can include tool arguments and task output. The read needs network access and does not run offline. Write to contact@embeddedriskanalytics.com for privacy and retention terms. + ## Capture the crew's events `FathomListener` records `tool_usage_finished`, `tool_usage_error`, and `task_completed` from the CrewAI event bus and writes them to a file when you close it. Keep a reference to the listener while the crew runs.