-
Notifications
You must be signed in to change notification settings - Fork 116
Add OpenRouter samples: prompt batch and budget gate #366
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
DABH
wants to merge
1
commit into
main
Choose a base branch
from
dabh/openrouter
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| import asyncio | ||
| import logging | ||
| import os | ||
| from datetime import timedelta | ||
|
|
||
| from agents import OpenAIProvider, set_tracing_disabled | ||
| from openai import AsyncOpenAI | ||
| from temporalio.client import Client | ||
| from temporalio.contrib.openai_agents import ModelActivityParameters, OpenAIAgentsPlugin | ||
| from temporalio.worker import Worker | ||
|
|
||
| from openai_agents.model_providers.workflows.openrouter_workflow import ( | ||
| OpenRouterAgentWorkflow, | ||
| ) | ||
|
|
||
|
|
||
| # @@@SNIPSTART python-openai-agents-openrouter-provider | ||
| def openrouter_provider() -> OpenAIProvider: | ||
| """OpenAI Agents SDK model provider backed by OpenRouter. | ||
|
|
||
| OpenRouter speaks the OpenAI Chat Completions API, so the stock provider | ||
| works once it is pointed at OpenRouter's base URL. Client retries are off: | ||
| the plugin runs each model call as a Temporal Activity, and Temporal owns | ||
| the retries. | ||
| """ | ||
| default_headers: dict[str, str] = {} | ||
| # Optional app attribution for OpenRouter's rankings. | ||
| if referer := os.getenv("OPENROUTER_HTTP_REFERER"): | ||
| default_headers["HTTP-Referer"] = referer | ||
| if title := os.getenv("OPENROUTER_APP_TITLE"): | ||
| default_headers["X-OpenRouter-Title"] = title | ||
|
|
||
| client = AsyncOpenAI( | ||
| base_url="https://openrouter.ai/api/v1", | ||
| api_key=os.environ["OPENROUTER_API_KEY"], | ||
| max_retries=0, | ||
| default_headers=default_headers or None, | ||
| ) | ||
| # Chat Completions is OpenRouter's primary endpoint; the Agents SDK | ||
| # defaults to the Responses API, which OpenRouter offers only in beta. | ||
| return OpenAIProvider(openai_client=client, use_responses=False) | ||
|
|
||
|
|
||
| # @@@SNIPEND | ||
|
|
||
|
|
||
| async def main(): | ||
| # Disable Agents SDK tracing: the default exporter sends traces to OpenAI's | ||
| # backend, which needs an OpenAI API key that this sample does not have. | ||
| set_tracing_disabled(disabled=True) | ||
|
|
||
| logging.basicConfig(level=logging.WARNING) | ||
| logging.getLogger("temporalio.workflow").setLevel(logging.DEBUG) | ||
|
|
||
| client = await Client.connect( | ||
| "localhost:7233", | ||
| plugins=[ | ||
| OpenAIAgentsPlugin( | ||
| model_params=ModelActivityParameters( | ||
| start_to_close_timeout=timedelta(seconds=60) | ||
| ), | ||
| model_provider=openrouter_provider(), | ||
| ), | ||
| ], | ||
| ) | ||
|
|
||
| worker = Worker( | ||
| client, | ||
| task_queue="openai-agents-model-providers-task-queue", | ||
| workflows=[ | ||
| OpenRouterAgentWorkflow, | ||
| ], | ||
| ) | ||
| await worker.run() | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| asyncio.run(main()) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| import asyncio | ||
|
|
||
| from temporalio.client import Client | ||
| from temporalio.contrib.openai_agents import OpenAIAgentsPlugin | ||
|
|
||
| from openai_agents.model_providers.workflows.openrouter_workflow import ( | ||
| OpenRouterAgentWorkflow, | ||
| ) | ||
|
|
||
|
|
||
| async def main(): | ||
| client = await Client.connect( | ||
| "localhost:7233", | ||
| plugins=[ | ||
| OpenAIAgentsPlugin(), | ||
| ], | ||
| ) | ||
|
|
||
| result = await client.execute_workflow( | ||
| OpenRouterAgentWorkflow.run, | ||
| "What's the weather in Tokyo?", | ||
| id="openai-agents-openrouter-workflow-id", | ||
| task_queue="openai-agents-model-providers-task-queue", | ||
| ) | ||
| print(f"Result: {result}") | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| asyncio.run(main()) |
28 changes: 28 additions & 0 deletions
28
openai_agents/model_providers/workflows/openrouter_workflow.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| from __future__ import annotations | ||
|
|
||
| from agents import Agent, Runner, function_tool | ||
| from temporalio import workflow | ||
|
|
||
| # Any OpenRouter model slug works here. A fixed, tool-capable model keeps the | ||
| # sample reproducible; swap in "openrouter/auto" to let OpenRouter choose. | ||
| OPENROUTER_MODEL = "openai/gpt-4o-mini" | ||
|
|
||
|
|
||
| @workflow.defn | ||
| class OpenRouterAgentWorkflow: | ||
| @workflow.run | ||
| async def run(self, prompt: str) -> str: | ||
| @function_tool | ||
| def get_weather(city: str): | ||
| workflow.logger.debug(f"Getting weather for {city}") | ||
| return f"The weather in {city} is sunny." | ||
|
|
||
| agent = Agent( | ||
| name="Assistant", | ||
| instructions="You only respond in haikus. When asked about the weather always use the tool to get the current weather.", | ||
| model=OPENROUTER_MODEL, | ||
| tools=[get_weather], | ||
| ) | ||
|
|
||
| result = await Runner.run(agent, prompt) | ||
| return result.final_output | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| # OpenRouter | ||
|
|
||
| These samples call [OpenRouter](https://openrouter.ai/) from Temporal Activities. OpenRouter serves hundreds of models from many providers behind one OpenAI-compatible API and one API key, and picks providers and models per request. Temporal handles everything around those calls: retries with backoff, fan-out with bounded concurrency, crash recovery, pausing for a human, and a durable per-attempt record of what was called and what it cost. | ||
|
|
||
| | Sample | Description | | ||
| |--------|-------------| | ||
| | [prompt_batch](prompt_batch) | Fan one OpenRouter call out per prompt with OpenRouter's Auto Router, and collect answer, model, and cost per prompt. Shows Temporal-owned retries, `Retry-After` handling, and retries served for free from OpenRouter's response cache. Start here. | | ||
| | [budget_gate](budget_gate) | The same batch, but it pauses instead of failing when money runs out, whether a soft budget in the Workflow or OpenRouter's own "insufficient credits" error, and resumes on a `raise_budget` Update. | | ||
|
|
||
| For OpenRouter as the model provider behind the [OpenAI Agents SDK plugin](../openai_agents), see [openai_agents/model_providers](../openai_agents/model_providers#openrouter). | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| 1. Follow the [repository prerequisites](../README.md), then install this sample's dependencies: | ||
|
|
||
| ```bash | ||
| uv sync --group openrouter | ||
| ``` | ||
|
|
||
| 2. Start a local dev server with the [Temporal CLI](https://docs.temporal.io/cli): | ||
|
|
||
| ```bash | ||
| temporal server start-dev | ||
| ``` | ||
|
|
||
| 3. Set an [OpenRouter API key](https://openrouter.ai/settings/keys) in the Worker's environment. A few cents of credit is enough for these samples. | ||
|
|
||
| ```bash | ||
| export OPENROUTER_API_KEY="sk-or-v1-..." | ||
| ``` | ||
|
|
||
| Optional: set `OPENROUTER_HTTP_REFERER` and `OPENROUTER_APP_TITLE` for [app attribution](https://openrouter.ai/docs/app-attribution) in OpenRouter's rankings. | ||
|
|
||
| The API key stays in the Worker process. Prompts, answers, models, and costs go through the Workflow and are recorded in Event History; the key never does. | ||
|
|
||
| ## Running a sample | ||
|
|
||
| Each sample has a Worker and a starter. Run them in separate terminals: | ||
|
|
||
| ```bash | ||
| # Terminal 1 | ||
| uv run --group openrouter openrouter/prompt_batch/run_worker.py | ||
|
|
||
| # Terminal 2 | ||
| uv run --group openrouter openrouter/prompt_batch/run_workflow.py "Explain retries in one sentence." "Write a haiku about databases." | ||
| ``` | ||
|
|
||
| ## How the Activity calls OpenRouter | ||
|
|
||
| [activities.py](activities.py) uses the `openai` SDK pointed at `https://openrouter.ai/api/v1`, which is the setup OpenRouter documents for OpenAI-compatible clients. OpenRouter-specific fields go in `extra_body`. Four things matter for durable execution: | ||
|
|
||
| - **Temporal owns retries.** The client is created with `max_retries=0`, so every attempt is one HTTP call and shows up in Event History. If you use OpenRouter's official `openrouter` package instead, pass `retry_config=RetryConfig("none", ...)`: by default it retries 5xx and connection errors for up to an hour, invisibly. | ||
| - **Errors are classified.** 408, 429, and 5xx raise a retryable `ApplicationError`; 400, 401, 402 (out of credits), 403 (moderation), and other 4xx raise a non-retryable one. A `Retry-After` header becomes the next retry delay. OpenRouter can also return HTTP 200 with an `error` body and no `choices`; the Activity checks for that. | ||
| - **Retries are free when the first call succeeded.** The Activity sends `X-OpenRouter-Cache: true`, so if a Worker dies after OpenRouter answered but before Temporal recorded the result, the retried, byte-identical request is served from OpenRouter's response cache and billed at $0. Nothing per-attempt goes in the request body, so attempts stay identical. | ||
| - **Heartbeats.** The Activity heartbeats so a dead Worker is detected after `heartbeat_timeout` (10s) rather than after the full `start_to_close_timeout`. | ||
|
|
||
| Each result carries the concrete model OpenRouter chose, OpenRouter's reported `usage.cost`, the generation id, and the cache status. | ||
|
|
||
| ## What Temporal does and does not guarantee | ||
|
|
||
| Activities are at-least-once. If a Worker dies mid-call, the retry re-sends the request; within the cache TTL that retry costs nothing, but two identical requests in flight at the same time both miss the cache and both bill. Completed Activities are never re-run, so a restarted batch resumes at the first unfinished prompt. | ||
|
|
||
| OpenRouter decides which provider and model serve a request, in milliseconds (Auto Router, `models` fallback lists, provider preferences). Temporal decides what happens over time: waiting out a rate limit, surviving a Worker crash, pausing for hours until a human acts, and keeping the audit trail. | ||
|
|
||
| ## Batch size | ||
|
|
||
| Each Activity adds a few events to the Workflow's Event History, and every answer is part of the Workflow result. These samples cap a batch at 100 prompts. For larger batches, use one Workflow per slice, or the pattern in [batch_sliding_window](../batch_sliding_window) with continue-as-new. | ||
|
|
||
| ## Tests | ||
|
|
||
| The tests replace OpenRouter with a fake HTTP transport and the Activity with a fake, so they need no API key and make no network calls: | ||
|
|
||
| ```bash | ||
| uv run --group openrouter pytest tests/openrouter | ||
| ``` |
Empty file.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we default to
openrouter/autosince the README saysand picks providers and models per request?