Skip to content

fix: reject replies from retired streamable http sessions - #1206

Open
nickcoai wants to merge 1 commit into
mainfrom
agent/streamable-http-session-origin
Open

nickcoai wants to merge 1 commit into
mainfrom
agent/streamable-http-session-origin

Conversation

@nickcoai

@nickcoai nickcoai commented Aug 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fix the late-handler reply issue from Dale's review.

This PR is based on main, independently of the worker refactor in #1205. The late-handler routing issue predates #1186. Whichever follow-up lands second will need a rebase to combine their transport changes.

  • Record the local transport and session generation when an inbound request is handed to the service.
  • Carry that origin through automatic responses and errors. Do not send an old handler's reply to a replacement session, including during shutdown draining.
  • Check handler identity when completing a request. If a new session reuses its request id, the old handler cannot remove or cancel the new one.
  • Keep the origin local. There are no wire-format changes or new public transport methods.

Scope

This protects automatic service replies, including request-association errors. Manually constructed raw transport replies have no captured origin.

Transport wrappers must preserve request extensions. Wrappers that defer the inner send within the same future retain the reply context; detached tasks must construct the inner send before spawning. Serializing requests or deferring send construction to a detached task loses the private context.

The tests cover late success and error replies, reused request ids, shutdown draining, and recovery during a pending reply send.

Validation

Rebased on main at 50fba5c2848d3ef9d0df0efacb8a7d001914f40e (October 17).
Exact head e575decaa047d9531fd918bb7afc61d074a0c9fa is green in CI, including all-features tests, non-local tests, clippy, MSRV, public API/semver checks, coverage, and examples. CodeQL also passed.

Local checks passed: 283 library tests with client,transport-streamable-http-client, all 24 streamable-http concurrency tests, nightly formatting, and git diff --check.

@github-actions github-actions Bot added T-test Testing related changes T-core Core library changes T-transport Transport layer changes labels Aug 23, 2026
@nickcoai
nickcoai marked this pull request as ready for review August 23, 2026 15:10
@nickcoai
nickcoai requested a review from a team as a code owner August 23, 2026 15:10
@nickcoai
nickcoai requested a review from DaleSeo August 23, 2026 15:10
Base automatically changed from agent/configurable-streamable-http-parallelism to main August 24, 2026 13:43
@nickcoai
nickcoai force-pushed the agent/streamable-http-session-origin branch from 6e6416c to 8477d25 Compare September 2, 2026 22:08
Comment thread crates/rmcp/src/service.rs Outdated
Comment thread crates/rmcp/src/service.rs
@nickcoai
nickcoai force-pushed the agent/streamable-http-session-origin branch from 8477d25 to ad88b6f Compare September 8, 2026 21:06

/// Check both the captured generation and, for replies, the originating transport.
#[cfg(feature = "transport-streamable-http-client")]
pub(crate) fn is_current_control(&self, request: &WorkerSendRequest<W>) -> bool {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this supposed to be visible only within the crate?

pub responder: tokio::sync::oneshot::Sender<Result<(), W::Error>>,
cancellation: Option<Arc<RequestCancellationRegistration>>,
control_generation: u64,
#[cfg(feature = "transport-streamable-http-client")]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the origin only be stored when transport-streamable-http-client is enabled?

@DaleSeo

DaleSeo commented Sep 9, 2026

Copy link
Copy Markdown
Member

Thanks for addressing my feedback, @nickcoai. I have a couple of questions before we move forward.

}
}

#[cfg(all(feature = "transport-worker", not(feature = "local")))]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we not want to run these tests with --all-features?

// Recovery may retire the origin between the initial check and the
// worker's dispatch check. Dropping that reply must not stop a drain
// from sending later, current-session replies.
if self.is_current() { result } else { Ok(()) }

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so the drain can tell "dropped because stale" from "the transport just broke".

Suggested change
if self.is_current() { result } else { Ok(()) }
match result {
Err(error) if !self.is_current() => {
tracing::debug!(%error, "discarding retired reply's send failure");
Ok(())
}
result => result,
}

@DaleSeo

DaleSeo commented Oct 5, 2026

Copy link
Copy Markdown
Member

This has merge conflicts with main now. Could you rebase onto the latest main and resolve them? Thanks!

@nickcoai
nickcoai force-pushed the agent/streamable-http-session-origin branch from ad88b6f to 205104e Compare October 9, 2026 23:20
Keep the inbound transport generation with automatic handler responses
and errors. Reject replies after session recovery, including during the
shutdown drain, without adding wire metadata.

Match the completing handler's token before removing its request id so
an old handler cannot cancel a replacement that reuses the id.

Cover late replies, id reuse, scoped send context, and shutdown races.
@nickcoai
nickcoai force-pushed the agent/streamable-http-session-origin branch from 205104e to e575dec Compare October 19, 2026 09:00
Comment on lines +1400 to +1404
let result = send.await;
// Recovery may retire the origin between the initial check and the
// worker's dispatch check. Dropping that reply must not stop a drain
// from sending later, current-session replies.
if self.is_current() { result } else { Ok(()) }

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once the origin retires, this turns every send error into Ok(()), which also hides real transport failures that happen during recovery. Could the streamable HTTP worker return Ok(()) for stale replies, like the stale branch in streamable_http_client.rs already does for cancellations? Then we could remove this re-check and its comment.

Suggested change
let result = send.await;
// Recovery may retire the origin between the initial check and the
// worker's dispatch check. Dropping that reply must not stop a drain
// from sending later, current-session replies.
if self.is_current() { result } else { Ok(()) }
send.await

Comment on lines +2057 to +2060
#[tokio::test]
async fn shutdown_drain_preserves_current_handler_after_old_response_or_error() {
for error in [false, true] {
for replacement_id in [8, 7] {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The nested loops put four scenarios in one test, with a 5-second timeout for each step. If one case hangs, it hides the others. Could we use rstest values to make each combination its own test?

Suggested change
#[tokio::test]
async fn shutdown_drain_preserves_current_handler_after_old_response_or_error() {
for error in [false, true] {
for replacement_id in [8, 7] {
#[rstest::rstest]
#[tokio::test]
async fn shutdown_drain_preserves_current_handler_after_old_response_or_error(
#[values(false, true)] error: bool,
#[values(8, 7)] replacement_id: i64,
) {
{
{

Comment on lines +1922 to +1924
#[test]
fn old_handler_completion_preserves_reused_id_owner() {
for error in [false, true] {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could use rstest here as well.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

T-core Core library changes T-test Testing related changes T-transport Transport layer changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants