-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Update compass doc #6125
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: develop
Are you sure you want to change the base?
Update compass doc #6125
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -1,27 +1,65 @@ | ||||||||||||||||||||||
| Training & Deploying COMPASS Navigation Policy with Real2Sim NuRec | ||||||||||||||||||||||
| ==================================================================== | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| COMPASS (Cross-embodiment Mobility Policy via Residual RL and Skill Synthesis) trains and | ||||||||||||||||||||||
| deploys cross-embodiment navigation policies on NuRec Real2Sim assets — photoreal | ||||||||||||||||||||||
| Gaussian-splat reconstructions of real spaces — in Isaac Lab. | ||||||||||||||||||||||
| COMPASS (Cross-embodiment Mobility Policy via Residual RL and Skill Synthesis) `[arXiv:2502.16372] <https://arxiv.org/abs/2502.16372>`_ trains cross-embodiment navigation policies. We support training COMPASS using NuRec assets in Isaac Lab. | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| This tutorial now lives in the **COMPASS handbook**, next to the code it documents, where it | ||||||||||||||||||||||
| stays in sync with the NuRec support branch. | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| NuRec Assets | ||||||||||||||||||||||
| ------------ | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| Real2Sim NuRec assets are available on `HuggingFace <https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-NuRec/tree/v0.2>`_. | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| Assets are generated by Nvidia's Real2Sim NuRec workflow. | ||||||||||||||||||||||
| Real2Sim NuRec pipeline generates data using multiple stereo RGB pipelines. | ||||||||||||||||||||||
| Examples of assets here. | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| Using 4 stereo RGB cameras | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| - `Galileo <https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-NuRec/tree/v0.2/nova_carter-galileo>`_ | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| .. figure:: ../../_static/policy_deployment/03_compass_with_NuRec/galileo.gif | ||||||||||||||||||||||
| :alt: Galileo NuRec Real2Sim asset | ||||||||||||||||||||||
| :width: 480px | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| Using 1 stereo RGB camera | ||||||||||||||||||||||
|
Comment on lines
+16
to
+24
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time! |
||||||||||||||||||||||
|
|
||||||||||||||||||||||
| - `Living Room <https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-NuRec/tree/v0.2/hand_hold-endeavor-livingroom>`_ | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| .. figure:: ../../_static/policy_deployment/03_compass_with_NuRec/living_room.gif | ||||||||||||||||||||||
| :alt: Living room NuRec Real2Sim asset | ||||||||||||||||||||||
| :width: 480px | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| We also support reconstruction done using `XGRIDS Portal Camera <https://www.xgrids.com/intl/portalcam>`_ using their own proprietary software. | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| - `Wormhole <https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-NuRec/tree/v0.2/xgrid-wormhole>`_ | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| .. figure:: ../../_static/policy_deployment/03_compass_with_NuRec/wormhole.gif | ||||||||||||||||||||||
| :alt: Wormhole Room XGRIDS reconstruction | ||||||||||||||||||||||
| :width: 480px | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| The complete tutorial for training COMPASS with NuRec assets is available in **COMPASS handbook**. | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| Where to find it | ||||||||||||||||||||||
| ---------------- | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| - **Repository:** `COMPASS on GitHub <https://github.com/NVlabs/COMPASS>`_ | ||||||||||||||||||||||
| - **Branch:** ``samc/support_nurec_assets_isaaclab_3.0`` (NuRec Real2Sim support) | ||||||||||||||||||||||
| - **Guide:** `docs/handbook/workflows/nurec_real2sim.md <https://github.com/NVlabs/COMPASS/blob/samc/support_nurec_assets_isaaclab_3.0/docs/handbook/workflows/nurec_real2sim.md>`_ in the COMPASS repository. | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| .. code-block:: bash | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| git clone https://github.com/NVlabs/COMPASS.git | ||||||||||||||||||||||
| cd COMPASS | ||||||||||||||||||||||
| git checkout samc/support_nurec_assets_isaaclab_3.0 | ||||||||||||||||||||||
| # Open docs/handbook/workflows/nurec_real2sim.md | ||||||||||||||||||||||
| - **Branch:** ``real2sim/isaaclab_3.0`` (NuRec asset support for training with COMPASS) | ||||||||||||||||||||||
| - **Guide:** `docs/handbook/workflows/nurec_real2sim.md <https://github.com/NVlabs/COMPASS/blob/real2sim/isaaclab_3.0/docs/handbook/workflows/nurec_real2sim.md>`_ in the COMPASS repository. | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| The guide covers Isaac Sim / Isaac Lab installation, COMPASS setup, downloading and | ||||||||||||||||||||||
| registering NuRec Real2Sim scenes, training a residual RL specialist, evaluation, | ||||||||||||||||||||||
| ONNX / TensorRT export, and ROS2 / sim-to-real deployment. | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| Sample training with NuRec assets with different embodiments. | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| 1. Galileo training with G1 robot | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| .. figure:: ../../_static/policy_deployment/03_compass_with_NuRec/galileo_g1_training.gif | ||||||||||||||||||||||
| :alt: Galileo G1 training view | ||||||||||||||||||||||
| :width: 480px | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| 2. Xgrid babyroom (along with sim obstacles) with Carter robot | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| .. figure:: ../../_static/policy_deployment/03_compass_with_NuRec/xgrid_wormhole_carter_training.gif | ||||||||||||||||||||||
| :alt: XGRIDS Wormhole training view | ||||||||||||||||||||||
| :width: 480px | ||||||||||||||||||||||
|
Comment on lines
+61
to
+65
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!