APP-17192: return per-file URLs from sequence dataset export - #882
Open
Vignesh P (vpandiarajan20) wants to merge 3 commits into
Open
APP-17192: return per-file URLs from sequence dataset export#882Vignesh P (vpandiarajan20) wants to merge 3 commits into
Vignesh P (vpandiarajan20) wants to merge 3 commits into
Conversation
The sequence dataset export produces three independent parquet files (sequences, tabular data, binary data). Packaging them into a single zip forces them to be serialized into one stream, so app must buffer two of the three fully in memory while the third is written. Returning one signed URL per file removes that constraint: each file can stream directly to object storage in parallel, making export memory flat regardless of dataset size. It also lets consumers that only need one file (e.g. training jobs reading tabular data) skip the other two. download_url is reserved rather than deprecated because it will never be populated again. Repo convention reserves fields that stop carrying data and deprecates only those that still work; this requires the 'allow breaking' label. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Vignesh P (vpandiarajan20)
force-pushed
the
app-17192-export-parquet-files
branch
from
July 22, 2026 17:20
d18a286 to
533cdec
Compare
Vignesh P (vpandiarajan20)
added
allow breaking
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
repeated SequenceDatasetExportFile filestoGetSequenceDatasetExportResponseand removes the singledownload_url.The export produces three independent parquet files, and zipping them forces serialization into one stream, so app currently buffers two of them fully in memory while the third is written:
domains/dataset/export.go#L61-L88(internal repo) shows the three generators each filling abytes.Bufferbefore the zip is assembled.🤖 Generated with Claude Code