Skip to content

fix: recreate grpc connection before retrying - #192

Closed
linkouth wants to merge 1 commit into
mainfrom
fix/recreate-grpc-connection-before-retry
Closed

fix: recreate grpc connection before retrying#192
linkouth wants to merge 1 commit into
mainfrom
fix/recreate-grpc-connection-before-retry

Conversation

@linkouth

@linkouth linkouth commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Since #180 the recreate and retry paths were mutually exclusive (shouldRecreateService && !shouldRetry), and #182 then added DEADLINE_EXCEEDED to the retryable codes. Combined, a wedged channel returning DEADLINE_EXCEEDED never got recreated while retries remained: the retry called getService(args), which returned the same cached client, so every attempt went down the same broken connection.

Recreate the client first and await it, so the retry builds a fresh connection. Other retryable codes still retry on the same client.

Recreation was also a silent no-op for reflection-based actions: reflectionServiceInstancesMap stores Promise<ServiceClient>, but clearInstancesCache compared that promise with the resolved client, so the check never matched and the cache was never cleared. Resolve the cached entry before comparing, and re-read the entry afterwards so a concurrent refreshCache cannot have its new client unset.

README documented the new behaviour already; its list of default retryable codes was missing DEADLINE_EXCEEDED.

Summary by Sourcery

Fix gRPC retries getting stuck on a broken connection by ensuring the client is fully recreated before a retry attempt.

Bug Fixes:

  • Fix retries on DEADLINE_EXCEEDED errors reusing the same wedged gRPC connection instead of a freshly recreated one.
  • Fix cache clearing for reflection-based gRPC clients, which previously never cleared the cache because promises were compared to resolved clients.
  • Guard against a concurrent cache refresh unsetting a newly created client while an older recreation was still in progress.
  • Add DEADLINE_EXCEEDED to the documented list of default retryable gRPC status codes.

Documentation:

  • Clarify README wording that the cached client is dropped before retrying, so the retry runs on a newly created connection while other retryable codes reuse the same connection.

Tests:

  • Add integration tests and test fixtures covering service recreation before retry for both direct and reflection-based gRPC clients.

Since #180 the recreate and retry paths were mutually exclusive
(`shouldRecreateService && !shouldRetry`), and #182 then added
DEADLINE_EXCEEDED to the retryable codes. Combined, a wedged channel
returning DEADLINE_EXCEEDED never got recreated while retries remained:
the retry called `getService(args)`, which returned the same cached
client, so every attempt went down the same broken connection.

Recreate the client first and await it, so the retry builds a fresh
connection. Other retryable codes still retry on the same client.

Recreation was also a silent no-op for reflection-based actions:
`reflectionServiceInstancesMap` stores `Promise<ServiceClient>`, but
`clearInstancesCache` compared that promise with the resolved client, so
the check never matched and the cache was never cleared. Resolve the
cached entry before comparing, and re-read the entry afterwards so a
concurrent `refreshCache` cannot have its new client unset.

README documented the new behaviour already; its list of default
retryable codes was missing DEADLINE_EXCEEDED.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@sourcery-ai

sourcery-ai Bot commented Aug 31, 2026

Copy link
Copy Markdown

🧙 Sourcery has finished reviewing your pull request!


Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've reviewed your changes and they look great!


Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@SeqviriouM

Copy link
Copy Markdown
Contributor

There is the similar PR: #190

@linkouth linkouth closed this Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants