Add bufferBeforeSend opt-in for reliable multipart upload retry with slow streaming sources#7030
Open
alextwoods wants to merge 3 commits into
Open
Add bufferBeforeSend opt-in for reliable multipart upload retry with slow streaming sources#7030alextwoods wants to merge 3 commits into
alextwoods wants to merge 3 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
fullBufferingEnabledopt-in for reliable multipart upload retry with slow streaming sourcesMotivation and Context
The S3 multipart upload pipeline has a structural defect when using
BufferedSplittableAsyncRequestBodywith slow streaming sources (e.g., SFTP). In the known-content-length path ofSplittingPublisher, each part body is emitted downstream (opening the HTTP connection) immediately upon initialization — before the part data is fully buffered. If the server returns a retryable error mid-stream, theRetryableSubAsyncRequestBodyretry buffer is null becausecomplete()has not yet been called, resulting in aNonRetryableException.This change adds an opt-in
fullBufferingEnabledoption that instructsSplittingPublisherto defer sending each part downstream until it is fully buffered — matching the behavior already used for the unknown-content-length path. The existing default behavior (immediate send) remains unchanged.Modifications
SplittingPublisher.java(internal)fullBufferingEnabledfield and correspondingBuilder.fullBufferingEnabled(boolean)method.initializeNextDownstreamBody():if (contentLengthKnown)→if (contentLengthKnown && !fullBufferingEnabled)to defer sending when full buffering is enabled.completeCurrentBody():if (upstreamSize == null && ...)→if ((upstreamSize == null || fullBufferingEnabled) && ...)to send after complete when full buffering is enabled.BufferedSplittableAsyncRequestBody.java(public API)private final boolean fullBufferingEnabledfield.create(AsyncRequestBody delegate, boolean fullBufferingEnabled)with Javadoc.splitCloseable()to passfullBufferingEnabledtoSplittingPublisher.Builder.create(AsyncRequestBody)passesfalse, preserving backward compatibility.Testing
Added new tests.
Types of changes
Checklist
mvn installsucceedsscripts/new-changescript and following the instructions. Commit the new file created by the script in.changes/next-releasewith your changes.License