-
Notifications
You must be signed in to change notification settings - Fork 8.4k
docs: add Fathom to Observability (Evaluation & Quality Assurance) #7247
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
ERA-Fathom
wants to merge
4
commits into
crewAIInc:main
Choose a base branch
from
ERA-Fathom:docs-fathom-observability
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
5426a8f
docs: add Fathom committed-state read to Observability
ERA-Fathom 8016342
docs: add Fathom card to Evaluation & Quality Assurance
ERA-Fathom 4e416c4
docs: preserve card indentation, add Fathom card
ERA-Fathom 6296466
docs: disclose hosted data transmission and correct coherent-run output
ERA-Fathom File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| --- | ||
| title: Fathom Committed-State Read | ||
| description: Check whether a crew's later actions contradict the records it already committed, from the events the crew already emits. | ||
| icon: check-double | ||
| mode: "wide" | ||
| --- | ||
|
|
||
| # Fathom Committed-State Read | ||
|
|
||
| ## Overview | ||
|
|
||
| [Fathom](https://github.com/ERA-Fathom/fathom) reads the tool calls and task completions a crew emits, folds the successful ones into the crew's committed state, and names the step where a later action contradicts an earlier commitment. That includes a record written against a key the crew already renamed, a task reported complete while its record was never written, an entity added to a collection that already holds it, and a value the crew replaced and then wrote again. The read is deterministic and needs no model access. It runs after the crew finishes, on a file you already have, and touches nothing in the crew's runtime path. | ||
|
|
||
| ## What it catches | ||
|
|
||
| | Finding | The crew... | | ||
| |---|---| | ||
| | `stale_reference` | acts on a record it already removed or renamed away | | ||
| | `superseded_value` | writes a value it already replaced | | ||
| | `residual` | ends the run with a record still carrying a value it replaced elsewhere | | ||
| | `duplicate_commit` | adds an entity a collection already holds | | ||
| | `post_commit_mutation` | changes a record after committing it | | ||
|
|
||
| A crew whose committed state stays consistent returns `coherent`. The read still prints the operation summary and then reports the committed state as coherent. | ||
|
|
||
| ## Installation | ||
|
|
||
| ```shell | ||
| uv add fathom-read | ||
| ``` | ||
|
|
||
| The package ships with a rate-limited demo key. For your own key, write to contact@embeddedriskanalytics.com and set `FATHOM_API_KEY`. | ||
|
|
||
| ## Data handling | ||
|
|
||
| `fathom read` sends the operation stream to the Fathom hosted read. That stream is derived from the crew's tool calls, so it can include tool arguments and task output. The read needs network access and does not run offline. Write to contact@embeddedriskanalytics.com for privacy and retention terms. | ||
|
|
||
| ## Capture the crew's events | ||
|
|
||
| `FathomListener` records `tool_usage_finished`, `tool_usage_error`, and `task_completed` from the CrewAI event bus and writes them to a file when you close it. Keep a reference to the listener while the crew runs. | ||
|
|
||
| ```python Code | ||
| from crewai import Agent, Task, Crew | ||
| from fathom_read.capture.crewai import FathomListener | ||
|
|
||
| listener = FathomListener("events.json") | ||
|
|
||
| crew = Crew(agents=[...], tasks=[...]) | ||
| result = crew.kickoff() | ||
|
|
||
| listener.close() | ||
| ``` | ||
|
|
||
| ## Run the read | ||
|
|
||
| ```shell | ||
| fathom read events.json --format crewai | ||
| ``` | ||
|
|
||
| If the crew's job involves a rename or a migration, tell the read which token replaced which, so it can also report records that still carry the old value when the run ends. | ||
|
|
||
| ```shell | ||
| fathom read events.json --format crewai --supersede guest_id=customer_id | ||
| ``` | ||
|
|
||
| `fathom read` exits 0 when the committed state is coherent and 2 when it finds a contradiction, so it drops into a test suite or a CI step as it is. Add `--json` for a machine-readable verdict, or `--ops` to see the action stream before anything is sent. | ||
|
|
||
| ## Map your own tools | ||
|
|
||
| The read knows common record-keeping tool names (`write_record`, `update_record`, `set_value`, `rename_key`, and others). For tools with your own names, map them once. | ||
|
|
||
| ```json tools.json | ||
| {"save_decision": {"op": "set", "kind": "decision", "key": "topic", "value": "text"}, | ||
| "book_seat": {"op": "add", "kind": "flight", "key": "seats", "value": "seat"}, | ||
| "confirm_booking": {"op": "commit", "kind": "flight", "key": "booking"}} | ||
| ``` | ||
|
|
||
| ```shell | ||
| fathom read events.json --format crewai --map tools.json | ||
| ``` | ||
|
|
||
| ## Example verdict | ||
|
|
||
| The [coherence census](https://github.com/ERA-Fathom/coherence-census) ships a minimal CrewAI event log in which a crew renames a key across three records, writes the third one on the old key, and reports the task complete. The read returns the following. | ||
|
|
||
| ```text | ||
| $ fathom read traces/crewai_events.json --format crewai --supersede guest_id=customer_id | ||
| ops read: 4 rejected (no-ops): 0 live facts: 4 | ||
| committed state: 1 finding | ||
| [residual] end of run: record 'r2' still carries 'guest_id' when the run ends; the agent replaced it with 'customer_id' elsewhere. | ||
| ``` | ||
|
|
||
| The full study, with sequential and hierarchical crews across 5, 12, and 20 coupled sub-records on two models, is at [Reading a crew's committed state from its own event stream](https://embeddedriskanalytics.com/research-reading-a-crews-committed-state-from-its-own-event-stream.html). | ||
|
|
||
| ## Resources | ||
|
|
||
| - [fathom-read on GitHub](https://github.com/ERA-Fathom/fathom) | ||
| - [Coherence census](https://github.com/ERA-Fathom/coherence-census), every framework the read has run on, with a trace per row | ||
| - [Embedded Risk Analytics research](https://embeddedriskanalytics.com/research.html) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.