[Proposal] Integrate Robocasa Scenes with Material and Articulation Support in IsaacLab

[makolon]

Proposal

Add support for Robocasa scenes by converting them to USD files and enabling their loading into IsaacLab. This proposal includes implementing a script to convert Robocasa's scenes and ensuring the ability to apply materials and maintain articulations within the scenes.

Additionally, Robocasa provides a detailed YAML file for each scene, which specifies the positions of various objects in MJCF format. To accurately recreate the Robocasa environment in Isaac Sim, we need to mimic these object placements based on the YAML file. Rather than converting the entire scene to USD at once using a USDExporter, we should convert individual objects to USD and then place them according to the YAML specifications. This will require implementing a script that follows the Isaac Lab API for object placement.

Motivation

The primary motivation for this proposal is to integrate Robocasa scenes into IsaacLab, providing more diverse and realistic environments for simulation. By converting Robocasa scenes to USD format and enabling the use of Robocasa's materials and articulations, this feature will enhance the flexibility and depth of simulation environments available in IsaacLab.

Using a YAML file to dictate object positions aligns with Robocasa's environment setup and allows us to maintain consistent object placements when converting to USD. This method ensures that the object placement script remains adaptable and follows the structure outlined in Isaac Lab's API.

Alternatives

An alternative approach could be to manually recreate the Robocasa scenes within IsaacLab. However, this would likely be time-consuming and may not fully capture the unique features of the original scenes. A single USD conversion of the entire scene could also be attempted, but it would limit the flexibility of object placement and customization as per the Robocasa YAML files.

Additional context

This proposal also includes enabling the conversion of articulated objects in Robocasa scenes while preserving their articulations in the USD format. Additionally, the proposal includes the capability to apply 12 distinct material styles provided by Robocasa via scripts and to create USD files without the robots present in the original scenes.

By adhering to the YAML file for object placement, we will ensure that Robocasa scenes are recreated with precision and are fully compatible with IsaacLab's ecosystem.

Checklist

• I have checked that there is no similar issue in the repo (required)

Acceptance Criteria

• Conversion of 10 Robocasa scenes to USD format.
• Implementation of a script that enables loading Robocasa scenes into IsaacLab.
• Application of 12 Robocasa material styles via scripting.
• Successful conversion of Robocasa articulated objects, retaining articulations in USD.
• Capability to generate USD files without robots in the scenes.
• Object placement script that follows the YAML specifications for accurate Robocasa scene replication.

References

• https://robocasa.ai/docs/tasks_scenes_assets/kitchen_scenes.html
• https://github.com/robocasa/robocasa/blob/main/robocasa/models/assets/scenes/kitchen_layouts/G_shaped_large.yaml
• https://isaac-sim.github.io/IsaacLab/source/api/lab/omni.isaac.lab.sim.converters.html
• https://github.com/isaac-sim/IsaacLab/tree/main/source/standalone/tools
• https://docs.omniverse.nvidia.com/isaacsim/latest/replicator_tutorials/tutorial_replicator_isaac_snippets.html
• https://isaac-sim.github.io/IsaacLab/source/api/lab/omni.isaac.lab.sim.spawners.html#omni.isaac.lab.sim.spawners.materials.VisualMaterialCfg

[makolon]

Progress

• I implemented the code to import the Robocasa scene by following the steps outlined below. However, it appears that the orientations of objects are not being properly exported to the JSON file. As a result, the scene generated by load_robocasa_scene.sh does not match the original.

Possible Causes

• During the scene-saving step, the function for retrieving object positions and orientations seems to be malfunctioning, causing the orientations to be saved as [0, 0, 0, 0].
• In the step for converting fixture assets to USD, the articulations may not be handled correctly. Alternatively, in Robocasa, articulated bodies are added after each object is spawned, which is not supported in Isaac Sim.

Procedures

1. Save the Robocasa scene parameters, such as object positions and orientations, to a JSON file. This step is performed inside the Robocasa Docker container, not in the IsaacLabExtendedTasks container. For details, refer to the Robocasa repository.

$ python3 -m robocasa.demos.demo_kitchen_scenes

2. Convert Objaverse assets to USD.

$ ./fix_objaverse_object_model.sh
$ ./convert_objaverse_object_mjcf.sh

3. Convert fixture assets to USD.

$ ./convert_fixture_mjcf.sh
$ ./fix_fixture_model.sh

3. Render the Robocasa scene in Omniverse Isaac Sim.

$ ./load_robocasa_scene.sh