From 1e9be69d0437403463e37ac6515002a2cc97c418 Mon Sep 17 00:00:00 2001 From: Evelyn Duesterwald Date: Tue, 28 Jul 2026 10:10:56 -0400 Subject: [PATCH 1/3] docs: document guideline generation modes and add end-to-end tutorial Consistency guidelines shipped without doc updates, so EVOLVE_GUIDELINES_MODE and its consistency-tuning env vars were undiscoverable anywhere in docs/. Adds a new Enabling Guidelines guide covering both regular and consistency modes, cross-links it from Configuration/Phoenix Sync/Low-Code Tracing, and reorders the Guides nav so tracing precedes sync (traces must exist before they can be synced). Also adds a new tutorial plus a companion example script that walks the full loop: trace an agent, generate guidelines from the trace, then retrieve and inject those guidelines back into a new run. Co-Authored-By: Claude Sonnet 5 --- .gitignore | 1 + docs/guides/configuration.md | 4 + docs/guides/extract-trajectories.md | 2 + docs/guides/guidelines.md | 71 ++++++++++++++ docs/guides/low-code-tracing.md | 2 + docs/guides/phoenix-sync.md | 9 +- docs/tutorials/guidelines-loop.md | 96 +++++++++++++++++++ docs/tutorials/index.md | 8 ++ .../low_code/guidelines_retrieval_demo.py | 85 ++++++++++++++++ mkdocs.yaml | 4 +- 10 files changed, 280 insertions(+), 2 deletions(-) create mode 100644 docs/guides/guidelines.md create mode 100644 docs/tutorials/guidelines-loop.md create mode 100644 examples/low_code/guidelines_retrieval_demo.py diff --git a/.gitignore b/.gitignore index 945607a9..440bd591 100644 --- a/.gitignore +++ b/.gitignore @@ -22,3 +22,4 @@ event.json site/ example-guidelines .codex +notes/ diff --git a/docs/guides/configuration.md b/docs/guides/configuration.md index 09a0b7ea..ee5f8ae4 100644 --- a/docs/guides/configuration.md +++ b/docs/guides/configuration.md @@ -45,6 +45,10 @@ All configuration variables are prefixed with `EVOLVE_`. |----------|-------------------------------------------------------------------------------|------------------------------------------| | `EVOLVE_BACKEND` | Backend provider (`milvus`, `filesystem`, or `postgres`) | `milvus` | | `EVOLVE_NAMESPACE_ID` | Namespace ID for isolation | `evolve` | +| `EVOLVE_GUIDELINES_MODE` | Guideline generation pipeline: `regular`, `consistency`, or `both` — see [Enabling Guidelines](guidelines.md) | `regular` | +| `EVOLVE_HIGH_UNCERTAINTY_THRESHOLD` | Consistency mode: steps scoring above this are treated as high-uncertainty — see [Enabling Guidelines](guidelines.md#configuring-consistency-guideline-generation) | `0.2` | +| `EVOLVE_LOW_UNCERTAINTY_THRESHOLD` | Consistency mode: steps scoring below this are treated as stable — see [Enabling Guidelines](guidelines.md#configuring-consistency-guideline-generation) | `0.1` | +| `EVOLVE_SKIP_ON_NO_UNCERTAINTY` | Consistency mode: skip guideline generation if no step exceeds the uncertainty threshold — see [Enabling Guidelines](guidelines.md#configuring-consistency-guideline-generation) | `true` | | `EVOLVE_GUIDELINES_MODEL` | Model for guideline generation only | `EVOLVE_MODEL_NAME` -> `gpt-4o` | | `EVOLVE_CONFLICT_RESOLUTION_MODEL` | Model for conflict resolution only | `EVOLVE_MODEL_NAME` -> `gpt-4o` | | `EVOLVE_FACT_EXTRACTION_MODEL` | Model for fact extraction only | `EVOLVE_MODEL_NAME` -> `gpt-4o` | diff --git a/docs/guides/extract-trajectories.md b/docs/guides/extract-trajectories.md index 3195b78b..20604074 100644 --- a/docs/guides/extract-trajectories.md +++ b/docs/guides/extract-trajectories.md @@ -2,6 +2,8 @@ A Python utility at `scripts/extract_trajectories.py` for extracting agent trajectories from Arize Phoenix traces and converting them to OpenAI chat completion message format. +This is a standalone export/debugging tool — it prints trajectories to stdout or a file and does not talk to Evolve or generate guidelines. To pull trajectories into Evolve *and* generate guidelines from them, use [Phoenix Sync](phoenix-sync.md) instead. + ## Features - Fetches spans from Phoenix's REST API with pagination support diff --git a/docs/guides/guidelines.md b/docs/guides/guidelines.md new file mode 100644 index 00000000..0f8e102a --- /dev/null +++ b/docs/guides/guidelines.md @@ -0,0 +1,71 @@ +# Enabling Guidelines + +Guidelines are short, actionable recommendations Evolve extracts from agent conversations ("trajectories") and stores as `guideline` entities. This guide covers how guideline generation works in **full Evolve (MCP server / CLI)** and how to choose between the two available generation methods, **regular** and **consistency**. + +> This guide applies to full Evolve. It does not apply to [Evolve Lite](../integrations/claude/evolve-lite.md), where guideline extraction happens entirely inside the host agent's own reasoning (a prompt-driven skill) rather than through the LLM pipeline described here. + +## Two ways trajectories reach the guideline pipeline + +| Entry point | When it runs | +|---|---| +| `save_trajectory` MCP tool | Called directly by an MCP client (e.g. Claude Desktop, Claude Code) at the end of a conversation | +| `evolve sync phoenix` CLI | Pulls previously-traced conversations out of Arize Phoenix in a batch — see [Phoenix Sync](phoenix-sync.md), which requires traces already flowing in via [Low-Code Tracing](low-code-tracing.md) | + +Both entry points funnel into the same underlying pipeline and respect the same `EVOLVE_GUIDELINES_MODE` setting below. + +## Choosing a guideline generation mode + +Set `EVOLVE_GUIDELINES_MODE` (or pass `--guidelines-mode` to `evolve sync phoenix`) to control which pipeline(s) run: + +| Mode | Optimizes for | What it does | +|---|---|---| +| `regular` (default) | Correctness on a single run | Single LLM pass over the trajectory; produces one guideline set. | +| `consistency` | Reliability across repeated runs | Resampling pass to score agent decision steps in the trajectory for consistency followed by a focused LLM pass to produce guidelines for inconsistent steps. | +| `both` | Both | Runs both pipelines and stores both sets of guidelines side by side. | + +```bash +# Regular guidelines (default) — no change needed +export EVOLVE_GUIDELINES_MODE=regular + +# Consistency guidelines only +export EVOLVE_GUIDELINES_MODE=consistency + +# Generate both +export EVOLVE_GUIDELINES_MODE=both +``` + +Or for a one-off Phoenix sync, set `--guidelines-mode` to `regular`, `consistency`, or `both`: + +```bash +uv run evolve sync phoenix --guidelines-mode consistency +``` + +### Configuring consistency guideline generation + +The consistency pipeline scores each decision step in a trajectory by resampling the decision multiple times and measuring how much the outcome varies, i.e. its uncertainty. The guideline-generation prompt is then steered toward the highest-uncertainty steps rather than summarizing the trajectory as a whole. + +Consistency guideline generation is noticeably more costly (multiple resample LLM calls per trajectory instead of one) and is worth it when you specifically want to catch agent behavior that's unstable across runs — decisions that the agent sometimes gets right and sometimes doesn't. + +Tunable via: + +| Variable | Default | Description | +|---|---|---| +| `EVOLVE_HIGH_UNCERTAINTY_THRESHOLD` | `0.2` | Steps scoring above this are treated as high-uncertainty | +| `EVOLVE_LOW_UNCERTAINTY_THRESHOLD` | `0.1` | Steps scoring below this are treated as stable | +| `EVOLVE_SKIP_ON_NO_UNCERTAINTY` | `true` | Skip guideline generation entirely if no step exceeds the uncertainty threshold | + +The resampling behavior itself (sample count, per-step-type uncertainty metric) is defined in a YAML config file shipped alongside the consistency pipeline; advanced users calling `generate_consistency_guidelines()` directly from Python can point it at a custom config via `config_path=`. + +## Verifying output + +```bash +uv run evolve entities list --type guideline +``` + +Each guideline's `metadata.generation_method` is `"regular"` or `"consistency"`, so you can tell which pipeline produced it when running in `both` mode. See [Guideline Provenance](low-code-tracing.md#6-understanding-guideline-provenance-metadata) for the full metadata schema, including `creation_mode` (`auto-mcp` vs `auto-phoenix` vs `manual`). + +## See also + +- [Configuration](configuration.md) — model selection (`EVOLVE_GUIDELINES_MODEL`) and other environment variables +- [Low-Code Tracing](low-code-tracing.md) — instrumenting your agent so traces reach Phoenix in the first place +- [Phoenix Sync](phoenix-sync.md) — batch guideline generation from traced trajectories diff --git a/docs/guides/low-code-tracing.md b/docs/guides/low-code-tracing.md index 86cbb500..dd4f452b 100644 --- a/docs/guides/low-code-tracing.md +++ b/docs/guides/low-code-tracing.md @@ -205,6 +205,8 @@ uv run evolve sync phoenix \ --include-errors ``` +See the [Phoenix Sync](phoenix-sync.md) guide for the full set of sync options, and [Enabling Guidelines](guidelines.md) for choosing between regular and consistency guideline generation. + ### 5. Verify Generated Guidelines ```bash diff --git a/docs/guides/phoenix-sync.md b/docs/guides/phoenix-sync.md index 049344d0..498ca1d1 100644 --- a/docs/guides/phoenix-sync.md +++ b/docs/guides/phoenix-sync.md @@ -2,6 +2,8 @@ Sync agent trajectories from Arize Phoenix to Evolve and automatically generate guidelines. +This guide assumes traces are already reaching Phoenix — see [Low-Code Tracing](low-code-tracing.md) to instrument your agent first. For choosing *how* guidelines get generated (regular vs. consistency), see [Enabling Guidelines](guidelines.md). + ## Overview The Phoenix sync module: @@ -44,13 +46,17 @@ uv run evolve sync phoenix \ --limit 500 \ --include-errors +# Generate consistency guidelines instead of regular ones +uv run evolve sync phoenix --guidelines-mode consistency + # Full options uv run evolve sync phoenix \ --url http://localhost:6006 \ --namespace production \ --project my_project \ --limit 200 \ - --include-errors + --include-errors \ + --guidelines-mode both ``` ### CLI Options @@ -62,6 +68,7 @@ uv run evolve sync phoenix \ | `--project` | `-p` | Phoenix project name | | `--limit` | | Max spans to fetch (default: 100) | | `--include-errors` | | Include failed/error spans | +| `--guidelines-mode` | | Guideline generation mode: `regular`, `consistency`, or `both` — see [Enabling Guidelines](guidelines.md) | ### Python API diff --git a/docs/tutorials/guidelines-loop.md b/docs/tutorials/guidelines-loop.md new file mode 100644 index 00000000..baa3ca5f --- /dev/null +++ b/docs/tutorials/guidelines-loop.md @@ -0,0 +1,96 @@ +# Starter Example: Evolving Agent Loop + +This tutorial takes you through the end-to-end evolving agent loop: an existing agent runs and is traced, guidelines are generated from that trace, and those guidelines are then fed back into a *new* run of the same agent, so it actually learns from what it did before. + +This tutorial covers **full Evolve (MCP server / CLI)**. It doesn't apply to [Evolve Lite](../integrations/claude/evolve-lite.md), where the equivalent loop (`/evolve-lite:learn`, then automatic injection on the next prompt) is handled entirely by the host agent — see the [Claude Code starter tutorial](../examples/hello_world/claude.md) for that version. + +!!! note "Set expectations" + The example agent here (`examples/low_code/smolagents_demo.py`) does simple arithmetic with two tools, so there's little room for it to behave meaningfully differently after "learning" — don't expect a dramatic before/after. The goal of this tutorial is to demonstrate the full wiring (trace → sync → generate → retrieve → inject) end to end on a small, fast example. Point the same steps at a real agent with real failure modes to see guidelines actually change behavior. + +## What you'll build + +You'll run a small pipeline end to end, in five stages: + +1. **Run** — a smolagents agent (`smolagents_demo.py`) executes a task, with tracing enabled. +2. **Trace** — that run is captured as a trajectory in Phoenix. +3. **Generate** — `evolve sync phoenix` pulls the trajectory out of Phoenix and generates guideline entities from it. +4. **Verify** — you'll confirm those guidelines were actually stored. +5. **Retrieve & inject** — a second script (`guidelines_retrieval_demo.py`) fetches the relevant guidelines and re-runs the same agent with them injected as instructions, closing the loop. + +By the end, you'll have watched a guideline travel all the way from "something the agent did" to "something the agent is told to do differently next time." + +## Requirements + +- [`uv` installed](https://docs.astral.sh/uv/getting-started/installation/) +- An LLM API key for your provider (e.g. `OPENAI_API_KEY`) + +Run every command below from the repo root (don't `cd` into `examples/low_code`, even to run the example scripts — Python adds a script's own directory to its import path automatically). This matters because the filesystem backend used throughout (`EVOLVE_BACKEND=filesystem`) stores data relative to the current directory; staying anchored at the repo root keeps every step reading and writing the same `evolve_data/`. + +## Step 0: Install dependencies + +```bash +uv sync --extra examples +``` + +Arize Phoenix and `sentence-transformers` (used for guideline retrieval) are core dependencies and come with any `altk-evolve` install. This step additionally pulls in `smolagents`, the OpenInference tracing instrumentors, and the example scripts' other dependencies. + +## Step 1: Start Phoenix + +```bash +uv run phoenix serve +# Server runs at http://localhost:6006 +``` + +## Step 2: Run the agent with tracing enabled + +```bash +EVOLVE_AUTO_ENABLED=true \ +EVOLVE_TRACING_PROJECT=guidelines-tutorial \ +uv run python examples/low_code/smolagents_demo.py +``` + +`smolagents_demo.py` runs a `CodeAgent` with two tools (`add`, `multiply`) against the task "What is (5 * 5) + 10?". With `EVOLVE_AUTO_ENABLED=true`, [Low-Code Tracing](../guides/low-code-tracing.md) patches the agent's LLM calls so the full trajectory reaches Phoenix. + +Run it two or three times so there's more than one trajectory to generate guidelines from. + +## Step 3: Sync the trace into Evolve and generate guidelines + +```bash +EVOLVE_BACKEND=filesystem \ +uv run evolve sync phoenix \ + --project guidelines-tutorial \ + --namespace guidelines-tutorial \ + --guidelines-mode regular +``` + +See [Phoenix Sync](../guides/phoenix-sync.md) for the full set of sync options, and [Enabling Guidelines](../guides/guidelines.md) if you want to try `--guidelines-mode consistency` or `both` instead. + +## Step 4: Verify guidelines exist + +```bash +EVOLVE_BACKEND=filesystem \ +uv run evolve entities list guidelines-tutorial --type guideline +``` + +You should see one or more `guideline` entities, each carrying `metadata.creation_mode: "auto-phoenix"` and `metadata.generation_method: "regular"`. + +## Step 5: Retrieve guidelines and re-run the agent with them injected + +```bash +EVOLVE_BACKEND=filesystem \ +uv run python examples/low_code/guidelines_retrieval_demo.py --namespace guidelines-tutorial --task "What is (5 * 5) + 10?" +``` + +This script: + +1. Calls `EvolveClient().select_guidelines(namespace_id, task_query)` — the same dosage-aware core + top-k retrieval used by the `get_relevant_guidelines` MCP tool — to fetch guidelines relevant to the task. +2. Formats the retrieved guideline content into a plain-text instructions block. +3. Passes that block as `instructions=` when constructing a fresh `CodeAgent`, then re-runs the same task. + +The printed output shows exactly which guidelines were retrieved and injected before the agent runs, so you can confirm the loop closed even though the toy task's final answer won't visibly change. + +## What's next + +- Point `EVOLVE_TRACING_PROJECT` / `--project` at a real agent with real tool failures and retry loops — that's where injected guidelines start visibly changing behavior. +- Swap `EVOLVE_BACKEND=filesystem` for Milvus or Postgres for persistent, semantic retrieval — see [Configuration](../guides/configuration.md). +- Try `--guidelines-mode consistency` in Step 3 to generate guidelines focused on the trajectory's least-stable steps instead — see [Enabling Guidelines](../guides/guidelines.md). diff --git a/docs/tutorials/index.md b/docs/tutorials/index.md index c25e35c0..851e1eac 100644 --- a/docs/tutorials/index.md +++ b/docs/tutorials/index.md @@ -29,3 +29,11 @@ Get started with Evolve on your platform of choice. Each tutorial walks through [Full tutorial →](../examples/hello_world/codex.md) + +--- + +## Starter Example: Evolving Agent Loop + +The starter examples above use Evolve Lite. This tutorial instead walks through **full Evolve (MCP server / CLI)**: taking an existing Python agent, tracing its runs, generating guidelines from those traces, and retrieving the guidelines back into a new run so the agent actually uses them. + +[Full tutorial →](guidelines-loop.md) diff --git a/examples/low_code/guidelines_retrieval_demo.py b/examples/low_code/guidelines_retrieval_demo.py new file mode 100644 index 00000000..22865acc --- /dev/null +++ b/examples/low_code/guidelines_retrieval_demo.py @@ -0,0 +1,85 @@ +import argparse +import os + +from altk_evolve.config.llm import llm_settings +from altk_evolve.frontend.client.evolve_client import EvolveClient + +from smolagents import CodeAgent, LiteLLMModel, tool + +# Reuse the same tools as smolagents_demo.py so this is a direct "part two" of that demo. +from local_mcp_server import add as mcp_add, multiply as mcp_multiply + + +@tool +def add(a: int, b: int) -> int: + """ + Add two numbers. + Args: + a: First number. + b: Second number. + """ + return mcp_add(a, b) # type: ignore[no-any-return,operator] + + +@tool +def multiply(a: int, b: int) -> int: + """ + Multiply two numbers. + Args: + a: First number. + b: Second number. + """ + return mcp_multiply(a, b) # type: ignore[no-any-return,operator] + + +def format_guidelines(selection) -> str: + """Turn retrieved guideline entities into a plain-text instructions block.""" + lines = [f"- {g.content}" for g in selection.all] + if not lines: + return "" + return "Apply these lessons learned from previous runs:\n" + "\n".join(lines) + + +def main(): + parser = argparse.ArgumentParser(description="Re-run the smolagents demo with retrieved guidelines injected.") + parser.add_argument( + "--namespace", + default=os.environ.get("EVOLVE_NAMESPACE_ID", "evolve"), + help="Namespace guidelines were synced/saved into", + ) + parser.add_argument("--task", default="What is (5 * 5) + 10?", help="Task to run the agent on") + args = parser.parse_args() + + client = EvolveClient() + selection = client.select_guidelines(namespace_id=args.namespace, task_query=args.task) + + print(f"Retrieved {len(selection.core)} core + {len(selection.retrieved)} task-specific guideline(s) from '{args.namespace}':") + for guideline in selection.all: + print(f" - {guideline.content}") + if not selection.all: + print(" (none found - run the guideline-generation steps first)") + + instructions = format_guidelines(selection) + + # Same model configuration pattern as smolagents_demo.py + model_id = os.environ.get("EVOLVE_EXAMPLE_AGENT_MODEL") or llm_settings.guidelines_model + custom_provider = llm_settings.custom_llm_provider + model = LiteLLMModel(model_id=model_id, custom_llm_provider=custom_provider) + + agent = CodeAgent( + tools=[add, multiply], + model=model, + add_base_tools=False, + instructions=instructions or None, + ) + + print("\nRunning Smolagents CodeAgent with guidelines injected as instructions...") + try: + result = agent.run(args.task) + print(f"Result: {result}") + except Exception as e: # noqa: BLE001 + print(f"Error running agent: {e}") + + +if __name__ == "__main__": + main() diff --git a/mkdocs.yaml b/mkdocs.yaml index 44e8f1e1..c942dcd5 100644 --- a/mkdocs.yaml +++ b/mkdocs.yaml @@ -78,11 +78,13 @@ nav: - Starter Example (IBM Bob): examples/hello_world/bob.md - Starter Example (Claude Code): examples/hello_world/claude.md - Starter Example (Codex): examples/hello_world/codex.md + - Starter Example (Evolving Agent Loop): tutorials/guidelines-loop.md - Guides: - Configuration: guides/configuration.md + - Enabling Guidelines: guides/guidelines.md + - Low-Code Tracing: guides/low-code-tracing.md - Phoenix Sync: guides/phoenix-sync.md - Extract Trajectories: guides/extract-trajectories.md - - Low-Code Tracing: guides/low-code-tracing.md - Memory Hooks: guides/memory-hooks.md - Data Retention: guides/retention.md - Reference: From 67f7b8140089f40bf2840ae3510474bc617dc870 Mon Sep 17 00:00:00 2001 From: Evelyn Duesterwald Date: Tue, 28 Jul 2026 16:12:22 -0400 Subject: [PATCH 2/3] fix(examples): propagate agent-run failures as a nonzero exit agent.run() failures were only printed, so the script exited 0 even when the run failed, hiding real failures from CI/automation. Re-raise as SystemExit, chained from the original exception, so the failure is actually visible in the exit status while keeping the error message. Addresses a CodeRabbit review comment on PR #297. Co-Authored-By: Claude Sonnet 5 --- examples/low_code/guidelines_retrieval_demo.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/low_code/guidelines_retrieval_demo.py b/examples/low_code/guidelines_retrieval_demo.py index 22865acc..e3f3e679 100644 --- a/examples/low_code/guidelines_retrieval_demo.py +++ b/examples/low_code/guidelines_retrieval_demo.py @@ -78,7 +78,7 @@ def main(): result = agent.run(args.task) print(f"Result: {result}") except Exception as e: # noqa: BLE001 - print(f"Error running agent: {e}") + raise SystemExit(f"Error running agent: {e}") from e if __name__ == "__main__": From c534c73bebf1d87f0c5a4e4895890680425bafd5 Mon Sep 17 00:00:00 2001 From: Evelyn Duesterwald Date: Wed, 29 Jul 2026 12:51:39 -0400 Subject: [PATCH 3/3] docs: mention both mode alongside regular/consistency in low-code-tracing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses a CodeRabbit nitpick on PR #297 — the sentence linking to Enabling Guidelines only mentioned regular and consistency, omitting the combined "both" mode. Co-Authored-By: Claude Sonnet 5 --- docs/guides/low-code-tracing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/low-code-tracing.md b/docs/guides/low-code-tracing.md index dd4f452b..2b626572 100644 --- a/docs/guides/low-code-tracing.md +++ b/docs/guides/low-code-tracing.md @@ -205,7 +205,7 @@ uv run evolve sync phoenix \ --include-errors ``` -See the [Phoenix Sync](phoenix-sync.md) guide for the full set of sync options, and [Enabling Guidelines](guidelines.md) for choosing between regular and consistency guideline generation. +See the [Phoenix Sync](phoenix-sync.md) guide for the full set of sync options, and [Enabling Guidelines](guidelines.md) for choosing between regular, consistency, or both guideline generation modes. ### 5. Verify Generated Guidelines