diff --git a/docs/source/_static/policy_deployment/03_compass_with_NuRec/galileo.gif b/docs/source/_static/policy_deployment/03_compass_with_NuRec/galileo.gif new file mode 100644 index 000000000000..ae49fc92b42d --- /dev/null +++ b/docs/source/_static/policy_deployment/03_compass_with_NuRec/galileo.gif @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1634907b792b7000b097bb272df46d33bab5a2c16d1b91df99268c833da5ad8d +size 10939128 diff --git a/docs/source/_static/policy_deployment/03_compass_with_NuRec/galileo_g1_training.gif b/docs/source/_static/policy_deployment/03_compass_with_NuRec/galileo_g1_training.gif new file mode 100644 index 000000000000..5b3f787b1d60 --- /dev/null +++ b/docs/source/_static/policy_deployment/03_compass_with_NuRec/galileo_g1_training.gif @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6cbbfe6ed78033cf6d3a25cd0cb9f2fb2028beee3e457b8050e8d96e9d23f5e2 +size 1614156 diff --git a/docs/source/_static/policy_deployment/03_compass_with_NuRec/living_room.gif b/docs/source/_static/policy_deployment/03_compass_with_NuRec/living_room.gif new file mode 100644 index 000000000000..47afb614a76e --- /dev/null +++ b/docs/source/_static/policy_deployment/03_compass_with_NuRec/living_room.gif @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:31d913f951537cee02194977ff65dd810afc3475b4962750a978108dbea7c1d1 +size 12303575 diff --git a/docs/source/_static/policy_deployment/03_compass_with_NuRec/wormhole.gif b/docs/source/_static/policy_deployment/03_compass_with_NuRec/wormhole.gif new file mode 100644 index 000000000000..99f70e2431bc --- /dev/null +++ b/docs/source/_static/policy_deployment/03_compass_with_NuRec/wormhole.gif @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b23ab980c9ead551433bbc469850528887e15d96b5c06be76e2c7deccc2b84c8 +size 6913566 diff --git a/docs/source/_static/policy_deployment/03_compass_with_NuRec/xgrid_wormhole_carter_training.gif b/docs/source/_static/policy_deployment/03_compass_with_NuRec/xgrid_wormhole_carter_training.gif new file mode 100644 index 000000000000..39b18b87466a --- /dev/null +++ b/docs/source/_static/policy_deployment/03_compass_with_NuRec/xgrid_wormhole_carter_training.gif @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b54c84195c8e688eac0df158c1ab8061ffdabff1a443de27e68a39c78dadc3fe +size 7521288 diff --git a/docs/source/policy_deployment/03_compass_with_NuRec/compass_navigation_policy_with_NuRec.rst b/docs/source/policy_deployment/03_compass_with_NuRec/compass_navigation_policy_with_NuRec.rst index 0f11d31bc7fd..c53c656adefd 100644 --- a/docs/source/policy_deployment/03_compass_with_NuRec/compass_navigation_policy_with_NuRec.rst +++ b/docs/source/policy_deployment/03_compass_with_NuRec/compass_navigation_policy_with_NuRec.rst @@ -1,27 +1,65 @@ Training & Deploying COMPASS Navigation Policy with Real2Sim NuRec ==================================================================== -COMPASS (Cross-embodiment Mobility Policy via Residual RL and Skill Synthesis) trains and -deploys cross-embodiment navigation policies on NuRec Real2Sim assets — photoreal -Gaussian-splat reconstructions of real spaces — in Isaac Lab. +COMPASS (Cross-embodiment Mobility Policy via Residual RL and Skill Synthesis) `[arXiv:2502.16372] `_ trains cross-embodiment navigation policies. We support training COMPASS using NuRec assets in Isaac Lab. -This tutorial now lives in the **COMPASS handbook**, next to the code it documents, where it -stays in sync with the NuRec support branch. + +NuRec Assets +------------ + +Real2Sim NuRec assets are available on `HuggingFace `_. + +Assets are generated by Nvidia's Real2Sim NuRec workflow. +Real2Sim NuRec pipeline generates data using multiple stereo RGB pipelines. +Examples of assets here. + +Using 4 stereo RGB cameras + + - `Galileo `_ + + .. figure:: ../../_static/policy_deployment/03_compass_with_NuRec/galileo.gif + :alt: Galileo NuRec Real2Sim asset + :width: 480px + +Using 1 stereo RGB camera + + - `Living Room `_ + + .. figure:: ../../_static/policy_deployment/03_compass_with_NuRec/living_room.gif + :alt: Living room NuRec Real2Sim asset + :width: 480px + +We also support reconstruction done using `XGRIDS Portal Camera `_ using their own proprietary software. + + - `Wormhole `_ + + .. figure:: ../../_static/policy_deployment/03_compass_with_NuRec/wormhole.gif + :alt: Wormhole Room XGRIDS reconstruction + :width: 480px + +The complete tutorial for training COMPASS with NuRec assets is available in **COMPASS handbook**. Where to find it ---------------- - **Repository:** `COMPASS on GitHub `_ -- **Branch:** ``samc/support_nurec_assets_isaaclab_3.0`` (NuRec Real2Sim support) -- **Guide:** `docs/handbook/workflows/nurec_real2sim.md `_ in the COMPASS repository. - -.. code-block:: bash - - git clone https://github.com/NVlabs/COMPASS.git - cd COMPASS - git checkout samc/support_nurec_assets_isaaclab_3.0 - # Open docs/handbook/workflows/nurec_real2sim.md +- **Branch:** ``real2sim/isaaclab_3.0`` (NuRec asset support for training with COMPASS) +- **Guide:** `docs/handbook/workflows/nurec_real2sim.md `_ in the COMPASS repository. The guide covers Isaac Sim / Isaac Lab installation, COMPASS setup, downloading and registering NuRec Real2Sim scenes, training a residual RL specialist, evaluation, ONNX / TensorRT export, and ROS2 / sim-to-real deployment. + +Sample training with NuRec assets with different embodiments. + +1. Galileo training with G1 robot + + .. figure:: ../../_static/policy_deployment/03_compass_with_NuRec/galileo_g1_training.gif + :alt: Galileo G1 training view + :width: 480px + +2. Xgrid babyroom (along with sim obstacles) with Carter robot + + .. figure:: ../../_static/policy_deployment/03_compass_with_NuRec/xgrid_wormhole_carter_training.gif + :alt: XGRIDS Wormhole training view + :width: 480px