Skip to content

[posthog-monitor] broker: end the session when the grant is revoked, don't poll it forever - #20

Merged
Automage merged 1 commit into
mainfrom
posthog-monitor/dead-grant-disconnect
Sep 4, 2026
Merged

[posthog-monitor] broker: end the session when the grant is revoked, don't poll it forever#20
Automage merged 1 commit into
mainfrom
posthog-monitor/dead-grant-disconnect

Conversation

@Automage

@Automage Automage commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Evidence

One install on 0.2.6 is emitting app_error / subsystem: broker / error_name: InvalidGrantError continuously — a tuple not seen once in the prior 14 days. The shape says loop, not incident:

started 11:58:05 UTC, still going at last check (16:11)
total so far 1,196 events over 4h13m
cadence one per poll — 360-369/hour when polling steadily, 60 of 60 minutes active
broker_offline emitted none — it isn't a network fault
status during still connected
reconnected no — no broker_connect_started, no broker_connected at any point

frames are entirely SDK-internal (@modelcontextprotocol/sdk/auth.js, streamableHttp.js:326), i.e. a refresh rejected with invalid_grant on a live client, not at connect time.

Not a 0.2.6 regression: the code path is identical in 0.2.5 (b6d7218). This is simply the first install whose refresh token died while telemetry was being watched.

Root cause

The app already knows what a dead grant means. isReauthRequired (robinhood/client.ts:175) matches exactly this, and connect() responds by dropping the tokens so the next Connect starts clean (:224-229). BrokerService.connect() then reflects it as disconnected "so the UI shows the Connect CTA (which re-consents)" (broker/index.ts:199-205).

That check exists only on the connect path — but the connect path is not where this surfaces. The stored token is accepted at startup and rejected on first use, so the failure lands in pollOnce's catch (broker/index.ts:367-376), which had three outcomes and no fourth for "this session is over":

  1. logPollFailure — deduped for host.log (its comment already names "a revoked grant" as the motivating case).
  2. isTransientNetworkErrorbroker_offline, deduped per outage.
  3. everything else → trackError, then poll again in 5-10s.

So the adapter keeps believing it is connected, the outage is never tracked, the panel keeps showing stale cached data, and nothing invites the user to reconnect. Left alone it does not stop: at the observed rate that is ~8,600 events per day from one install.

The change

pollOnce's catch now ends a definitively dead session, reaching the same state connect() already reaches:

if (isTransientNetworkError(err)) this.noteOffline(err);
else {
  analytics.trackError("broker", err, "caught", brokerErrorCode(err));
  if (isDeadGrantError(err)) await this.endDeadSession();
}
  • isDeadGrantError joins the other broker error classifiers in network-error.ts, keeping the SDK import out of the adapter-agnostic service.
  • endDeadSession() delegates to disconnect() — session forgotten, polling stopped, status disconnected. Dropping the tokens is the point: keeping a revoked grant would leave isAuthorized() true and invite a silent reconnect that can only fail the same way.
  • Silent by design. The user finds it disconnected the next time they look and clicks Connect. Notifying sooner is a separate product decision and is deliberately not in this PR.

Scope is narrower than isReauthRequired on purpose. That predicate also counts a resource UnauthorizedError, which is right when deciding whether to start a consent the user just asked for, and wrong on the poll path — a transient or endpoint-specific 401 would throw away credentials that still work. invalid_grant is the authorization server's permanent verdict; a 401 is not. Both behaviours are pinned by tests.

Side effect worth noting: the loop stopping means this fault emits 1 app_error instead of 1,196 and counting, without touching the deliberate "a non-network failure is still an app_error — every time" invariant — a mapping bug thrown from our own code still reports on every poll, as intended.

Verification

  • bun install --frozen-lockfile — clean.
  • bun run typecheck — clean.
  • bunx @biomejs/biome check on the three changed files — clean. (There is no lint script in app/package.json; biome is the linter.)
  • bun test from app/: 366 pass / 7 fail with this change vs 364 pass / 7 fail on the same base without it (measured by stashing). The +2 are the new tests; the 7 failures (AgentRegistry CLAUDE.md composition ×2, CodexClient app-server WebSocket ×5) are identical either way and unrelated to this diff. Note bun test must run from app/ — the @shared/* alias doesn't resolve from the repo root.

Two tests added to connect.test.ts, using the existing pollScript harness: a revoked grant reports once then leaves status: disconnected / isAuthorized(): false with further polls silent; a mid-session 401 reports and stays connected.

Risk

Low, and confined to one recovery path. No change to spawning, connecting, or the transient-outage path. The failure mode if the classification were ever wrong is a disconnect the user resolves with one click — and it is gated on the single error class whose meaning is "this credential is permanently void".

Branch note: this went to posthog-monitor/dead-grant-disconnect rather than my usual branch, which currently carries open PR #15 — pushing there would have silently rewritten that PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_0128hXeESThVH1BrjUp2MbUP


Review fixes (folded into the single commit)

Reviewed and taken over by @pranav. The diagnosis above stands; the routine's implementation had one blocking bug and a few gaps, fixed and squashed into this one commit (a follow-up simplification review then removed a speculative throw guard around the token drop):

  • Self-deadlock (blocking). endDeadSession() called disconnect(), which awaits inflight. When the revoked grant surfaces on the first poll — which runConnect awaits before startPolling(), while inflight still holds runConnect — that await waited on itself: connect() never resolved, status stuck on connected, and Reset hung on the same await. Reproduced with a test (hung on the first commit, resolves on main). Now the teardown is inline (forgetSession, shared with disconnect()), and runConnect skips startPolling() if the first poll ended the session.
  • Telemetry. The automatic end now emits broker_session_ended {reason: invalid_grant} after the single app_error, so a stranded install is countable instead of looking like one error that recovered.
  • Tests. FakeAdapter had no hasTokens(), so the isAuthorized() assertion was vacuous (it passed with the session end stubbed out). It now tracks tokens through succeed()/reset(); the test also asserts adapter.resets. A new test pins the deadlock regression.

Verification on the branch: bun test from app/ 374 pass / 0 fail, bun run typecheck clean, biome clean on the changed files. docs/ARCHITECTURE.md (§6.6, §12.3) and docs/TODO.md are updated in the worktree copy and will be reconciled into the main checkout on merge.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 3, 2026

Copy link
Copy Markdown

Deploying opentradeoss with  Cloudflare Pages  Cloudflare Pages

Latest commit: 4c74907
Status: ✅  Deploy successful!
Preview URL: https://0f9c705c.opentradeoss.pages.dev
Branch Preview URL: https://posthog-monitor-dead-grant-d.opentradeoss.pages.dev

View logs

@Automage Automage self-assigned this Sep 3, 2026
@Automage
Automage force-pushed the posthog-monitor/dead-grant-disconnect branch from 2707a3e to dd69764 Compare September 4, 2026 13:12
One install on 0.2.6 emitted app_error {broker, InvalidGrantError} 167 times in 26
minutes — one per poll — with status still `connected`, no broker_offline, and no
reconnect afterwards. The stored access token was honoured at startup and the refresh
was rejected later, on a live client, so the failure landed in pollOnce's catch, which
only knew "transient → broker_offline" and "else app_error, poll again in 5–10 s".
`isReauthRequired` already handles a dead grant, but only on the connect path.

pollOnce's catch now ends a definitively dead session: `isDeadGrantError`
(InvalidGrantError only — deliberately narrower than isReauthRequired, since a
mid-session 401 must not cost the user credentials that still work) → `forgetSession`:
polling stopped, per-session state cleared, status `disconnected` so the Connect CTA
comes back, tokens dropped — then `broker_session_ended {reason: invalid_grant}`, a
lifecycle event beside broker_offline/online so the funnel can count stranded installs.
One app_error per revoked grant instead of one per poll; the "a non-network failure is
still an app_error — every time" invariant is untouched for everything else.

`forgetSession` is the teardown shared with disconnect(), and the poll path must call
it rather than disconnect() itself: that awaits `inflight`, and the dead grant can
surface on the first poll, which runConnect awaits while `inflight` still holds
runConnect — the await would wait on itself forever (status stuck on `connected`, Reset
hung too). runConnect skips startPolling() if that first poll ended the session, and
forgetSession drops the tokens last so status is settled before the one DB step.

Tests (connect.test.ts): FakeAdapter gets hasTokens() tied to succeed()/reset() so
isAuthorized() assertions are real; a revoked grant reports once, ends the session, and
further polls are silent; a mid-session 401 keeps the session; a revoked grant on the
connect-time first poll resolves connect() (the deadlock regression) and leaves no
poller.
@Automage
Automage force-pushed the posthog-monitor/dead-grant-disconnect branch from dd69764 to 4c74907 Compare September 4, 2026 13:19
@Automage
Automage merged commit 6d42c75 into main Sep 4, 2026
1 check passed
@Automage
Automage deleted the posthog-monitor/dead-grant-disconnect branch September 4, 2026 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant