fix: recreate grpc connection before retrying - #192
Closed
linkouth wants to merge 1 commit into
Closed
Conversation
Since #180 the recreate and retry paths were mutually exclusive (`shouldRecreateService && !shouldRetry`), and #182 then added DEADLINE_EXCEEDED to the retryable codes. Combined, a wedged channel returning DEADLINE_EXCEEDED never got recreated while retries remained: the retry called `getService(args)`, which returned the same cached client, so every attempt went down the same broken connection. Recreate the client first and await it, so the retry builds a fresh connection. Other retryable codes still retry on the same client. Recreation was also a silent no-op for reflection-based actions: `reflectionServiceInstancesMap` stores `Promise<ServiceClient>`, but `clearInstancesCache` compared that promise with the resolved client, so the check never matched and the cache was never cleared. Resolve the cached entry before comparing, and re-read the entry afterwards so a concurrent `refreshCache` cannot have its new client unset. README documented the new behaviour already; its list of default retryable codes was missing DEADLINE_EXCEEDED. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
🧙 Sourcery has finished reviewing your pull request! Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Contributor
|
There is the similar PR: #190 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since #180 the recreate and retry paths were mutually exclusive (
shouldRecreateService && !shouldRetry), and #182 then added DEADLINE_EXCEEDED to the retryable codes. Combined, a wedged channel returning DEADLINE_EXCEEDED never got recreated while retries remained: the retry calledgetService(args), which returned the same cached client, so every attempt went down the same broken connection.Recreate the client first and await it, so the retry builds a fresh connection. Other retryable codes still retry on the same client.
Recreation was also a silent no-op for reflection-based actions:
reflectionServiceInstancesMapstoresPromise<ServiceClient>, butclearInstancesCachecompared that promise with the resolved client, so the check never matched and the cache was never cleared. Resolve the cached entry before comparing, and re-read the entry afterwards so a concurrentrefreshCachecannot have its new client unset.README documented the new behaviour already; its list of default retryable codes was missing DEADLINE_EXCEEDED.
Summary by Sourcery
Fix gRPC retries getting stuck on a broken connection by ensuring the client is fully recreated before a retry attempt.
Bug Fixes:
Documentation:
Tests: