Inquiry about the release timeline for VLABench evaluation pipeline

[daoshi1593]

Summary

First of all, thank you for the fantastic work on VLABench! I'm writing to inquire about the estimated release date for the evaluation pipeline and documentation.

Context

Some of my research focuses on evaluating various Vision-Language-Action (VLA) models, with a particular interest in those based on the OpenVLA architecture.

VLABench, especially its emphasis on challenging long-horizon tasks and instruction following, seems like the perfect platform for my work. I am very excited to test how different models perform on it.

The Question

The announcement mentions that a "complete infra framework" including "VLABench evaluation" will be open-sourced. I'm very eager to get started, but I couldn't find detailed instructions in the current preview.

Could you please share an estimated timeline for when these evaluation details and the corresponding code will be publicly available?

This information would be incredibly helpful for planning my experiments.

Thank you again for your contribution to the community. Looking forward to the full release!

[Shiduo-zh]

Thanks for your question! The infrastructure in this repo is tied to my ongoing pre-training work. I plan to release the updated version along with a series of new projects, and I expect this to happen around December. At that time, I’ll also update the instructions accordingly. Before that, you can refer to https://github.com/Shiduo-zh/openpi for efficient evaluation using GPU parallelism.

Thanks for your patience!