[PB-5902]: Add global retry with backoff to HttpClient for rate limit handling

[terrerox]

Adds a retryWithBackoff utility that automatically retries HTTP requests when a 429 – Too Many
Requests response is received.

Key points

• Retries are based on the retry-after response header.
• Integrated into all HttpClient HTTP methods via an execute wrapper.
• Retry is disabled by default — enable it globally through HttpClient.enableGlobalRetry, or
  per-service by passing retryOptions inside the ApiSecurity object.
• Per-service options take priority over global options.

Behavior details

• Retries only on 429 Too Many Requests.
• Delay is read from the retry-after header (seconds).
• Delay is capped at maxRetryAfter (default: 70000ms) regardless of header value.
• If the header is missing or invalid, the error is thrown immediately.
• After all retries are exhausted, the original error is re-thrown with a Max retries exceeded
  prefix.
• Setting maxRetries to 0 in enableGlobalRetry is a compile-time error via the NonZero type
  constraint.

Configuration examples

// Per-service: only Storage retries
Storage.client(apiUrl, appDetails, { token, retryOptions: { maxRetries: 3 } }); 
Auth.client(apiUrl, appDetails); // no retry

// Global: all services retry
HttpClient.enableGlobalRetry({ maxRetries: 5 });

// Per-service overrides global
HttpClient.enableGlobalRetry({ maxRetries: 5 }); 
Storage.client(apiUrl, appDetails, { token, retryOptions: { maxRetries: 2 } }); // uses 2

[CandelR]

If I am not mistaken, this adds the retry as mandatory to all requests. Modify it so that it is disabled by default and can be implemented by projects that want it, as it can cause problems.
Add more tests to check different cases.

[CandelR]

Maybe we will need a retry after limit to avoid clients wait infinite time. When I was checking the backend rate limits on mobile, there was a server failure and Cloudflare returned a 6-month reset. Perhaps a default maximum time should be added (which should also be configurable) to avoid this potential error

[CandelR]

this will produce an additional retry. The retries number is maxRetries+1 if I'm not mistaken

[terrerox]

Mmm, nope, because the condition is attempt < opts.maxRetries, not attempt <= opts.maxRetries. With this, the loop will finish with one try left, and that’s where it happens

[CandelR]

outside of the loop you have added an additional call. That means that in the worst case scenario, it is called maxretrires +1.
Although now that I look back, I hadn't realised that the default call, which is not a retry, is included here. It seems a bit strange to me to see it implemented this way, as it gives the impression that all calls are retries.
If the retries are at 0, it does not enter the loop and executes the one outside; if they are at 1, it enters the loop, executes the first one, and if everything goes well, it finishes, and if the server returns a 429, the next iteration of the loop is not executed and it does the ‘retry’ outside.
But maybe that's just my impression and it's fine as it is. If you all agree, I won't say anything else

[terrerox]

You’re right, that’s something I didn’t realize until now, so I’ve changed my approach. What do you think?

[larryrider]

If I am not mistaken, this adds the retry as mandatory to all requests. Modify it so that it is disabled by default and can be implemented by projects that want it, as it can cause problems. Add more tests to check different cases.

which projects wont need it? @CandelR For me, its better if its enabled by default (or at least, is easy to enable)
for example if this works as expected, its a win-win for the cli as i dont have to do another implementation, i will use this one

maybe i miss something but this retry-mechanism looks good and is needed for everyone right?

[CandelR]

If I am not mistaken, this adds the retry as mandatory to all requests. Modify it so that it is disabled by default and can be implemented by projects that want it, as it can cause problems. Add more tests to check different cases.

which projects wont need it? @CandelR For me, its better if its enabled by default (or at least, is easy to enable) for example if this works as expected, its a win-win for the cli as i dont have to do another implementation, i will use this one

maybe i miss something but this retry-mechanism looks good and is needed for everyone right?

In mobile, for example, retry is already implemented. If it is added here silently by default in a minor update and someone does not notice, it could cause unwanted behaviour. The entire implementation would have to be removed if the SDK is updated.
Apart from that, I think that since it is a library shared by different projects, features should not be added silently and forced.
If the situation were different, I wouldn't say I wouldn't leave it as the default, but I think that in our case it's not the best option, at least initially.
I think it's simpler and better to add a flag that can be activated if desired in each case

[larryrider]

If I am not mistaken, this adds the retry as mandatory to all requests. Modify it so that it is disabled by default and can be implemented by projects that want it, as it can cause problems. Add more tests to check different cases.

which projects wont need it? @CandelR For me, its better if its enabled by default (or at least, is easy to enable) for example if this works as expected, its a win-win for the cli as i dont have to do another implementation, i will use this one
maybe i miss something but this retry-mechanism looks good and is needed for everyone right?

In mobile, for example, retry is already implemented. If it is added here silently by default in a minor update and someone does not notice, it could cause unwanted behaviour. The entire implementation would have to be removed if the SDK is updated. Apart from that, I think that since it is a library shared by different projects, features should not be added silently and forced. If the situation were different, I wouldn't say I wouldn't leave it as the default, but I think that in our case it's not the best option, at least initially. I think it's simpler and better to add a flag that can be activated if desired in each case

Then we should know first which projects will be affected to decide if it affects more enabled or disabled by default. We should also not make this change silent in a patch version, maybe we should also discuss it on the drive channel so everyone knows about this (and change this in a minor version).

In any case I agree with you that we would need an easy way to enable (or disable) it, maybe a new optional property at the ApiSecurity for example

[terrerox]

If I am not mistaken, this adds the retry as mandatory to all requests. Modify it so that it is disabled by default and can be implemented by projects that want it, as it can cause problems. Add more tests to check different cases.

which projects wont need it? @CandelR For me, its better if its enabled by default (or at least, is easy to enable) for example if this works as expected, its a win-win for the cli as i dont have to do another implementation, i will use this one
maybe i miss something but this retry-mechanism looks good and is needed for everyone right?

In mobile, for example, retry is already implemented. If it is added here silently by default in a minor update and someone does not notice, it could cause unwanted behaviour. The entire implementation would have to be removed if the SDK is updated. Apart from that, I think that since it is a library shared by different projects, features should not be added silently and forced. If the situation were different, I wouldn't say I wouldn't leave it as the default, but I think that in our case it's not the best option, at least initially. I think it's simpler and better to add a flag that can be activated if desired in each case

Then we should know first which projects will be affected to decide if it affects more enabled or disabled by default. We should also not make this change silent in a patch version, maybe we should also discuss it on the drive channel so everyone knows about this (and change this in a minor version).

In any case I agree with you that we would need an easy way to enable (or disable) it, maybe a new optional property at the ApiSecurity for example

The globalRetry option is now disabled by default. See the documentation above for details 🚀

[larryrider]

hey @terrerox have you tried to enable it on any project, like drive-web for example?

I dont see the ApiSecurity property and the way to enable it on every sdk service (auth, users, etc).

It could be also (maybe) a good idea to not make static the retryOptions from the httpclient so it can be enabled or disabled per every sdk service. And then in every project (cli, drive-web, etc) we can enable it individually or globally

[terrerox]

hey @terrerox have you tried to enable it on any project, like drive-web for example?

I dont see the ApiSecurity property and the way to enable it on every sdk service (auth, users, etc).

It could be also (maybe) a good idea to not make static the retryOptions from the httpclient so it can be enabled or disabled per every sdk service. And then in every project (cli, drive-web, etc) we can enable it individually or globally

I think it’s necessary and useful to have static retryOptions so they can be used as a fallback when service-level retries are not configured. It would work like this:

// Per-service: only Storage retries
Storage.client(apiUrl, appDetails, { token, retryOptions: { maxRetries: 3 } }); 
Auth.client(apiUrl, appDetails); // no retry

// Global: all services retry
HttpClient.enableGlobalRetry({ maxRetries: 5 });

// Per-service overrides global
HttpClient.enableGlobalRetry({ maxRetries: 5 }); 
Storage.client(apiUrl, appDetails, { token, retryOptions: { maxRetries: 2 } }); // uses 2