[DataLoader] Support concurrent multi-file reads per split#539
Merged
robreeves merged 9 commits intolinkedin:mainfrom Apr 13, 2026
Merged
[DataLoader] Support concurrent multi-file reads per split#539robreeves merged 9 commits intolinkedin:mainfrom
robreeves merged 9 commits intolinkedin:mainfrom
Conversation
…iceberg Re-introduce the ArrivalOrder scan order and batch_size parameter that were removed in linkedin#504. The original removal was necessary because the fork dependency (sumedhsakdeo/iceberg-python) could not pass ELR. Now that li-pyiceberg 0.11.3 includes the ArrivalOrder API from upstream (apache/iceberg-python#3046), we can restore the functionality using an approved registry dependency.
…per_split Add files_per_split parameter to OpenHouseDataLoader that controls how many files each DataLoaderSplit reads concurrently. DataLoaderSplit now accepts a list of FileScanTasks and sets concurrent_streams to match, enabling parallel I/O within a single split.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Builds on #537. Adds a
files_per_splitparameter toOpenHouseDataLoaderthat controls how many files eachDataLoaderSplitreads concurrently.DataLoaderSplitnow accepts a list ofFileScanTasks and setsconcurrent_streamsto match, enabling parallel I/O within a single split. Defaults to 1 (preserving current behavior).Changes
data_loader.py— Newfiles_per_splitparameter (default 1). Scan tasks are grouped into chunks before being passed toDataLoaderSplit.data_loader_split.py—__init__now takesfile_scan_tasks: Sequence[FileScanTask]instead of a single task.concurrent_streamsis set to match the number of files. Theidproperty hashes all file paths for determinism.Tests — New multi-file split tests in
test_data_loader_split.py(iteration, deterministic IDs, transforms, empty-list validation). Newfiles_per_splittests intest_data_loader.py(grouping, remainder, larger-than-total, data preservation, invalid input).Testing Done
make verifypasses — 213 tests pass, lint, format, and mypy all green.Additional Information