Embedded auth runtime DCR does not bind upstream refresh tokens to the DCR client generation

[aron-muon]

Summary

The embedded auth server persists upstream provider refresh tokens without recording which upstream DCR client generated them. If a DCR-sensitive provider setting changes (most visibly scopes), ToolHive resolves a new DCR client but keeps existing user refresh tokens. Later refreshes present those old tokens with the current client, and the provider rejects them with invalid_grant / client_id mismatch.

This leaves every grant from the previous client generation latent-broken until its access token expires.

Verified on ToolHive v0.44.0. I also checked v0.46.0 and current main: storage.UpstreamTokens still has no upstream-client binding field, and upstreamTokenRefresher still refreshes through the currently resolved provider client.

Reproduction

1. Configure an embedded-auth OAuth2 upstream using runtime DCR:

upstreamProviders:
  - name: example
    type: oauth2
    oauth2Config:
      dcrConfig:
        registrationEndpoint: https://provider.example/register
      scopes: [read]

2. Authorize a user and retain the upstream refresh token.
3. Change scopes to [read, write] and restart/reconcile the proxy.
4. The DCR cache key changes because it includes ScopesHash, so ToolHive registers a new upstream OAuth client.
5. Let the user's old access token expire and make a request.
6. ToolHive calls the provider token endpoint with the old refresh token and the new client ID. Providers that bind refresh tokens to clients reject it.

Observed provider error:

upstream token refresh failed: token request failed: invalid_grant - client_id mismatch

Root cause

pkg/authserver/server/handlers/callback.go stores:

storageTokens := &storage.UpstreamTokens{
    // ...
    ClientID: pending.ClientID,
}

That ClientID is the downstream client (for example LibreChat), not the upstream DCR client used by BaseOAuth2Provider for the authorization-code exchange.

pkg/authserver/refresher.go later selects the provider by ProviderID and calls:

provider.RefreshTokens(ctx, expired.RefreshToken, expired.UpstreamSubject)

The provider contains the current DCR-resolved oauth2.Config.ClientID. Nothing verifies that it matches the client that minted expired.RefreshToken.

There is a related reauthorization hazard: maybeCarryForwardRefreshToken can copy the latest prior refresh token into a newly authorized session when the provider omits a new refresh token, without checking its upstream DCR generation. That can reintroduce an obsolete refresh token immediately after a successful reconnect.

Fleet evidence

One deployment accumulated three upstream client generations for the same provider:

• original statically configured DCR client: 92 sessions
• first runtime-DCR client (read): 28 sessions
• current runtime-DCR client (read + write): 79 sessions

The provider's refresh tokens are JWTs containing the public client_id claim, so classification is exact: 120 stale sessions versus 79 current sessions. Every observed client_id mismatch belonged to one of the stale generations.

The DCR client IDs themselves are long-lived; this was caused by config generations, not client expiration.

Other providers may issue opaque refresh tokens, making the same drift impossible to classify after the fact.

Expected behavior

An upstream grant should be bound to the exact upstream client/config generation that obtained it. Before refresh, ToolHive should either:

1. refresh with that same upstream client while it remains registered, or
2. detect a generation mismatch, delete/invalidate the stale provider grant, and return a typed reauthorization-required error without calling the token endpoint.

Proposed fix

• Add an upstream binding field to UpstreamTokens, such as UpstreamClientID or a stable DCR/config-generation fingerprint.
• Populate it from the resolved upstream provider during callback storage.
• Validate it before refresh.
• Validate it before maybeCarryForwardRefreshToken reuses a prior refresh token.
• On mismatch, remove the stale provider row and return a typed reauthorization-required result rather than retrying invalid_grant.
• Add provider/user-scoped revocation or migration APIs so operators do not need to manipulate Redis storage directly.
• Add tests for:
  • scope changes
  • static-client to runtime-DCR migration
  • redirect URI / registration endpoint changes
  • reauthorization where the provider omits a new refresh token
  • opaque refresh-token providers

Operational impact

The failure is delayed until access-token expiry, so a rollout can look healthy for hours or days while every pre-change user grant is already doomed. Operators need a controlled reconnect migration every time a DCR-sensitive field changes until the binding is represented in storage.

[aron-muon]

Additional verified fleet data and operational cleanup results:

• Production had 199 upstream Grafana grant rows: 120 tied to obsolete upstream clients, 79 tied to the current client.
• Obsolete generations were independently identifiable from the refresh token's public JWT client_id claim:
  • original static client: 92 rows
  • first runtime-DCR (grafana:read): 28 rows
  • current runtime-DCR (grafana:read + grafana:write): 79 rows
• All six observed client_id mismatch failures belonged to obsolete generations.
• Access-token scopes were not a reliable generation discriminator: 14 current-client sessions had only the provider-granted query + read subset.
• A controlled cleanup removed only the 120 obsolete upstream provider rows, their session/user reverse-index memberships, and 124 downstream refresh-token sessions whose serialized Fosite session referenced those exact upstream tsid values. Verification afterward: 79 current upstream rows remain, zero obsolete upstream rows remain, zero downstream refresh sessions reference an obsolete tsid, and the current DCR row is preserved.

This also exposed an important migration concern for opaque-refresh-token providers: after a DCR-sensitive config change there is no way to classify old grants from token contents. The durable fix needs the upstream client/config generation persisted alongside UpstreamTokens, not provider-specific token introspection.