[python] Add LeRobot Dataset v3 import - #9446
Draft
XiaoHongbo-Hope wants to merge 5 commits into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Add a one-time PyPaimon importer:
LeRobot Dataset v3 -> Arrow batches -> Paimon tableThis follows
load_from_hdf5()and reuses the existing connection, table creation, strict append validation, BLOB, batch write, and snapshot commit paths. It does not add dynamic training conversion, Dataset Version, Ray, Torch DataLoader, or Paimon-to-LeRobot export.Public API:
source_options, and a non-path string may be a Hugging Facerepo_id.meta/info.json; an existing table uses the same strict schema validation and append behavior as HDF5 import.feature_mappingandtransformare optional.Feature mapping:
Images retain their compressed source bytes. Video is decoded through the official LeRobot API and each frame is stored once as an independent PNG BLOB, so the complete MP4 is not copied into every row. URI metadata and Parquet files are read directly through Paimon FileIO. Because the LeRobot 0.4 video decoder requires a local path, only the active MP4 chunk for each video feature is cached temporarily and removed after import; credentials are never inherited from the target Catalog.
LeRobot remains optional through
pypaimon[lerobot]. The first version supports LeRobot Dataset v3 and rejects v2.x,uint64, depth video, and unsupported language event structures with explicit errors.Tests
multimodal_lerobot_test.py: 9 passedmultimodal_hdf5_test.py: 24 passedmultimodal_table_test.py: 68 passed./paimon-python/dev/lint-python.sh -i license,flake8: passedgit diff --check: passed