Skip to content

feat: add non-buffered response streaming#2082

Open
GrahamCampbell wants to merge 2 commits into
async-aws:masterfrom
GrahamCampbell:response-buffer-streaming
Open

feat: add non-buffered response streaming#2082
GrahamCampbell wants to merge 2 commits into
async-aws:masterfrom
GrahamCampbell:response-buffer-streaming

Conversation

@GrahamCampbell

@GrahamCampbell GrahamCampbell commented May 3, 2026

Copy link
Copy Markdown
Contributor

Allow streaming output operations to opt out of response buffering so large downloads can be consumed without temp storage while keeping buffered behavior as the default. My use case is streaming a large CSV file out of S3. Without this change, I have to go around the SDK and create a presigned URL and download it myself using another HTTP client. Without the ability to turn off buffering, the entire file gets written to /tmp.

Allow streaming output operations to opt out of response buffering so large downloads can be consumed without temp storage while keeping buffered behavior as the default.
Regenerate service inputs to document @responseBuffer consistently and bump affected service packages to the core version that supports the option.
@GrahamCampbell

GrahamCampbell commented May 3, 2026

Copy link
Copy Markdown
Contributor Author

Hmmm, tough to test due to the cyclic nature of needing a newer version of core...

@jderusse

jderusse commented May 4, 2026

Copy link
Copy Markdown
Member

I would avoid creating precedence by adding a new custom @responseBuffer parameter.

I didn't yet read the implementation, but some replacement suggestion in mind

  1. Is there a reason for buffering the output? Couldn't we alway stream without buffer? (rewind come in my mind, but do we have something else? could we let end-user buffer the output if they want to rewind. And release a new Major version that don't buffer)
  2. or buffer when the response is small, and that size is configurable with something like new S3Client(['doNotBufferWhenSizeExceed' => '3Gib'])
  3. Or maybe more simple : can't we add a new response->getContentAsResource(buffer: false) to achieve the same?

@GrahamCampbell

Copy link
Copy Markdown
Contributor Author

Yes, we do want to buffer the output in general by default (and by default, small responses buffer entirely in memory without spilling to disk anyway). Without buffering it, we can't read it more than once, and there are many situations where we would want to do that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants