feat: add --user-prefix for per-agent sticky session routing - #22
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
Adds optional per-agent sticky-session identity support to batchbench-agent by introducing a --user-prefix flag that stamps each request with a stable OpenAI user field (<prefix>-<agent_id>), enabling router-level session affinity and better prefix-cache locality for agent loops.
Changes:
- Added
AgentLoopConfig.user_prefix: Option<String>plus awith_user_prefixbuilder. - Updated agent request construction to optionally include the OpenAI
userfield using the agent id. - Wired
--user-prefixthrough the agent CLI and added targeted tests for default omission vs. stamping behavior.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| rust/src/agent.rs | Adds user_prefix to agent loop config and stamps request bodies with user when configured; adds tests. |
| rust/src/agent_cli.rs | Exposes --user-prefix and threads it into AgentLoopConfig. |
Comment on lines
+223
to
+226
| pub fn with_user_prefix(mut self, user_prefix: impl Into<String>) -> Self { | ||
| self.user_prefix = Some(user_prefix.into()); | ||
| self | ||
| } |
Comment on lines
+383
to
+385
| if let Some(user_prefix) = &args.user_prefix { | ||
| config = config.with_user_prefix(user_prefix.clone()); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an optional
--user-prefixflag to agent mode. When set, every request body carries an OpenAIuserfield of"<prefix>-<agent_id>", stable across all turns of an agent's loop.Why: gateway/router layers that implement session-sticky routing (e.g. keying a routing header off the typed
userfield) need a per-session identity to pin each agent's growing conversation to one backend worker, so prefix-cache reuse actually happens. Without it, all agents either share one static header identity or have none, and multi-rank deployments round-robin the turns — destroying the cache locality that agent mode exists to measure.Default behaviour is unchanged (
useris omitted unless the flag is passed).Changes
AgentLoopConfig.user_prefix: Option<String>+with_user_prefixbuilderbuild_request_bodytakes theagent_idand stampsuserwhen configured--user-prefixCLI flag wired throughagent_cli"<prefix>-<agent_id>"when setTesting
cargo test --lib agent— 15 passed (2 new assertions/tests).🤖 Generated with Claude Code
https://claude.ai/code/session_01GhkeGWXgQY9x9P5c6wzwv5