feat(grpc-gcp): penalize retryable channel errors

[rahul2393]

With dynamic scaling enabled, a channel returning retryable errors (UNAVAILABLE / RESOURCE_EXHAUSTED) kept its low active-stream count and so kept attracting picks: the pool had no signal that a channel was unhealthy until its streams actually piled up, and scale-up never triggered for a pool that was failing rather than busy.

Change

• Error penalty on retryable failures. A call completing with UNAVAILABLE or RESOURCE_EXHAUSTED adds errorPenaltyStep (new pool option) of synthetic load to that channel for errorPenaltyDuration, capped at maxRpcPerChannel. The picker sees the penalty through getPickerLoad(), steering new picks toward healthy channels; expiry is lazy and lock-free.
• Scale-up integration. Penalty load counts toward the dynamic scale-up trigger, so a pool that is failing-but-idle can still grow.
• Single net-delta aggregate accounting. Every update to the shared penalty aggregate is computed under the channel lock and applied as one atomic delta, so lock-free readers never observe an intermediate state; a channel leaving the pool clears its contribution.
• Negative errorPenalty duration disables the penalty

[gemini-code-assist]

Code Review

This pull request introduces an error penalty mechanism for channels experiencing retryable errors (such as UNAVAILABLE or RESOURCE_EXHAUSTED). It adds configuration options for the penalty step and duration, tracks individual and aggregate penalty loads, and incorporates these penalties into the channel load calculations and scale-up signaling. Additionally, a comprehensive test suite has been added to verify the correctness of the error penalty logic under various scenarios. I have no feedback to provide.

[olavloite]

This does not take any error penalty into account. For read/write transactions, that is maybe (probably) the right choice. But for read-only transactions, where channel affinity is just a 'nice-to-have', it is probably not the right choice.

[rahul2393]

yes but that means we need to send transaction type info here which will be beyond the scope of this PR, can be done in follow-up PR

[olavloite]

Should we indiscriminately apply errors for all RESOURCE_EXHAUSTED errors? Or only the ones that include certain metadata? (e.g. a RetryInfo)

[rahul2393]

In design review it was agreed that we will apply for all RESOURCE_EXHAUSTED errors so want to keep same here

[olavloite]

LGTM, except for the configuration option setters.

(Also do a recheck whether the PR description is up to date with the implementation)

[rahul2393]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces an error penalty mechanism to the GCP Managed Channel pool. When retryable errors (such as UNAVAILABLE or RESOURCE_EXHAUSTED) occur on a channel, a decaying load penalty is applied to that channel. This penalty is added to the active stream count to compute the "picker load", which is then used by the channel selection strategies to steer traffic away from degraded channels. The PR also adds configuration options for the penalty step and duration, updates the scale-up and scale-down logic to account for these penalties, and includes comprehensive unit tests. There are no review comments, so I have no feedback to provide.