Skip to content

[python] Add paired HDF5/Paimon ACT benchmark - #9466

Draft
YannByron wants to merge 3 commits into
apache:masterfrom
YannByron:m0/pr3-act-benchmark
Draft

[python] Add paired HDF5/Paimon ACT benchmark#9466
YannByron wants to merge 3 commits into
apache:masterfrom
YannByron:m0/pr3-act-benchmark

Conversation

@YannByron

@YannByron YannByron commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Add a paired ACT benchmark that compares the original RoboMIND AgileX HDF5 path with the Paimon path under one deterministic training contract. Both backends share the same configuration, train-only normalization, random seed, window sequence, ACT policy, optimizer, and correctness gates.

Changes

  • Add a snapshot-pinned, lazy ContiguousWindowDataset for map-style PyTorch access to complete row windows.
  • Coalesce overlapping rows during batched access and support anchor-only columns so observation BLOBs are loaded once per action window.
  • Add HDF5 and Paimon ACT adapters plus a shared deterministic CPU harness with tensor and loss parity checks.
  • Alternate backend execution for at least three rounds and report first-batch latency, DataLoader throughput, fixed-step time, and separately measured Python allocations.
  • Reuse the RoboMIND pipeline's versioned canonical-action statistics and reject source, snapshot, normalization, or window-sequence mismatches.
  • Add act and hdf5 optional dependencies, CLI guidance, and PyTorch/window API documentation.
  • Fall back to UNKNOWN source provenance when the installed package is not inside a Git checkout.

Testing

  • PR3 plus packed-video conflict regression (paired_act_benchmark, contiguous_window_dataset, RoboMIND, Ray HDF5, multimodal video/table, and Torch tests): 144 passed, 1 skipped
  • Non-E2E Python suite before the latest master rebase (4523 passed, 134 skipped; one unrelated Ray shuffle ordering assertion failed once and passed on isolated rerun)
  • Isolated Ray shuffle rerun (1 passed)
  • Ruff on all changed Python files
  • Python license checks (707 files valid)
  • compileall and git diff --check
  • Wheel build and content check for the benchmark and window-dataset modules
  • Full ACT benchmark on a downloaded RoboMIND dataset (requires dataset and LeRobot model assets)

Notes

This builds directly on the RoboMIND AgileX HDF5 pipeline merged in #9445. Ingestion and canonical-action backfill are intentionally outside the measured benchmark scope.

YannByron and others added 3 commits August 29, 2026 22:50
Add a generic snapshot-pinned window Dataset that builds a lightweight row-id index, reads projected payloads on demand, and handles continuity, tail policies, transforms, and multi-worker DataLoader use.

Co-Authored-By: Codex <noreply@anthropic.com>

Co-Authored-By: Codex <noreply@openai.com>

AI-Model: gpt-5.6-sol
Co-Authored-By: Codex <noreply@ai-tool.com>
AI-Contributed/Feature: 2/2
AI-Contributed/UT: 0/0
Run both backends through one deterministic ACT harness and pin lazy Paimon windows to the normalization snapshot. Adapt the benchmark to the rebased lazy source contract, coalesce overlapping batch reads, and measure Python allocations outside wall-clock timing.

Co-Authored-By: Codex <noreply@anthropic.com>

Co-Authored-By: Codex <noreply@openai.com>

AI-Model: gpt-5.6-sol
Co-Authored-By: Codex <noreply@ai-tool.com>
AI-Contributed/Feature: 1418/1418
AI-Contributed/UT: 363/363
Read observation images only at each window anchor and tolerate installed packages without a Git checkout. Improve public option documentation and CLI guidance.

Co-Authored-By: Codex <noreply@openai.com>

AI-Model: gpt-5
AI-Contributed/Feature: 200/200
AI-Contributed/UT: 25/25
@YannByron
YannByron force-pushed the m0/pr3-act-benchmark branch from 622ceae to 5d8890e Compare August 29, 2026 14:53
@YannByron
YannByron marked this pull request as draft August 29, 2026 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant