Skip to content

Add bufferBeforeSend opt-in for reliable multipart upload retry with slow streaming sources#7030

Open
alextwoods wants to merge 3 commits into
masterfrom
alexwoo/buffer_before_send
Open

Add bufferBeforeSend opt-in for reliable multipart upload retry with slow streaming sources#7030
alextwoods wants to merge 3 commits into
masterfrom
alexwoo/buffer_before_send

Conversation

@alextwoods

Copy link
Copy Markdown
Contributor

Add fullBufferingEnabled opt-in for reliable multipart upload retry with slow streaming sources

Motivation and Context

The S3 multipart upload pipeline has a structural defect when using BufferedSplittableAsyncRequestBody with slow streaming sources (e.g., SFTP). In the known-content-length path of SplittingPublisher, each part body is emitted downstream (opening the HTTP connection) immediately upon initialization — before the part data is fully buffered. If the server returns a retryable error mid-stream, the RetryableSubAsyncRequestBody retry buffer is null because complete() has not yet been called, resulting in a NonRetryableException.

This change adds an opt-in fullBufferingEnabled option that instructs SplittingPublisher to defer sending each part downstream until it is fully buffered — matching the behavior already used for the unknown-content-length path. The existing default behavior (immediate send) remains unchanged.

Modifications

SplittingPublisher.java (internal)

  • Added fullBufferingEnabled field and corresponding Builder.fullBufferingEnabled(boolean) method.
  • Changed initializeNextDownstreamBody(): if (contentLengthKnown)if (contentLengthKnown && !fullBufferingEnabled) to defer sending when full buffering is enabled.
  • Changed completeCurrentBody(): if (upstreamSize == null && ...)if ((upstreamSize == null || fullBufferingEnabled) && ...) to send after complete when full buffering is enabled.

BufferedSplittableAsyncRequestBody.java (public API)

  • Added private final boolean fullBufferingEnabled field.
  • Added new factory method: create(AsyncRequestBody delegate, boolean fullBufferingEnabled) with Javadoc.
  • Updated splitCloseable() to pass fullBufferingEnabled to SplittingPublisher.Builder.
  • Existing create(AsyncRequestBody) passes false, preserving backward compatibility.

Testing

Added new tests.

Types of changes

  • [] Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)

Checklist

  • I have read the CONTRIBUTING document
  • Local run of mvn install succeeds
  • My code follows the code style of this project
  • My change requires a change to the Javadoc documentation
  • I have updated the Javadoc documentation accordingly
  • I have added tests to cover my changes
  • All new and existing tests passed
  • I have added a changelog entry. Adding a new entry must be accomplished by running the scripts/new-change script and following the instructions. Commit the new file created by the script in .changes/next-release with your changes.
  • My change is to implement 1.11 parity feature and I have updated LaunchChangelog

License

  • I confirm that this pull request can be released under the Apache 2 license

@alextwoods alextwoods requested a review from a team as a code owner June 11, 2026 17:08
@alextwoods alextwoods changed the title Add fullBufferingEnabled opt-in for reliable multipart upload retry with slow streaming sources Add bufferBeforeSend opt-in for reliable multipart upload retry with slow streaming sources Jun 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant