robosuite v1.3 release #260

yukezhu · 2021-10-19T04:52:47Z

robosuite 1.3.0 Release Notes

Highlights
New Features
Improvements
Critical Bug Fixes
Other Bug Fixes

Highlights

This release of robosuite brings powerful rendering functionalities including new renderers and multiple vision modalities, in addition to some general-purpose camera utilties. Below, we discuss the key details of these new features:

Renderers

In addition to the native Mujoco renderer, we present two new renderers, NVISII and iGibson, and introduce a standardized rendering interface API to enable easy swapping of renderers.

NVISII is a high-fidelity ray-tracing renderer originally developed by NVIDIA, and adapted for plug-and-play usage in robosuite. It is primarily used for training perception models and visualizing results in high quality. It can run at up to ~0.5 fps using a GTX 1080Ti GPU. Note that NVISII must be installed (pip install nvisii) in order to use this renderer.

iGibson is a much faster renderer that additionally supports physics-based rendering (PBR) and direct rendering to pytorch tensors. While not as high-fidelity as NVISII, it is incredibly fast and can run at up to ~1500 fps using a GTX 1080Ti GPU. Note that iGibson must be installed (pip install igibson) in order to use this renderer.

With the addition of these new renderers, we also introduce a standardized renderer for easy usage and customization of the various renderers. During each environment step, the renderer updates its internal state by calling update() and renders by calling render(...). The resulting visual observations can be polled by calling get_pixel_obs() or by calling other methods specific to individual renderers. We provide a demo script for testing each new renderer, and our docs also provide additional information on specific renderer details and installation procedures.

Vision Modalities

In addition to new renderers, we also provide broad support for multiple vision modalities across all (Mujoco, NVISII, iGibson) renderers:

RGB: Standard 3-channel color frames with values in range [0, 255]. This is set during environment construction with the use_camera_obs argument.
Depth: 1-channel frame with normalized values in range [0, 1]. This is set during environment construction with the camera_depths argument.
Segmentation: 1-channel frames with pixel values corresponding to integer IDs for various objects. Segmentation can occur by class, instance, or geom, and is set during environment construction with the camera_segmentations argument.

In addition to the above modalities, the following modalities are supported by a subset of renderers:

Surface Normals: [NVISII, iGibson] 3-channel (x,y,z) normalized direction vectors.
Texture Coordinates: [NVISII] 3-channel (x,y,z) coordinate texture mappings for each element
Texture Positioning: [NVISII, iGibson] 3-channel (x,y,z) global coordinates of each pixel.

Specific modalities can be set during environment and renderer construction. We provide a demo script for testing the different modalities supported by NVISII and a demo script for testing the different modalities supported by iGibson.

Camera Utilities

We provide a set of general-purpose camera utilities that intended to enable easy manipulation of environment cameras. Of note, we include transform utilities for mapping between pixel, camera, and world frames, and include a CameraMover class for dynamically moving a camera during simulation, which can be used for many purposes such as the DemoPlaybackCameraMover subclass that enables smooth visualization during demonstration playback.

Improvements

The following briefly describes other changes that improve on the pre-existing structure. This is not an exhaustive list, but a highlighted list of changes.

Standardize EEF frames (fixed osc position controller bug [updated] #204). Now, all grippers have identical conventions for plug-and-play usage across types.
Add OSC_POSITION control option for spacemouse (Add OSC_POSITION option in spacemouse input_utils #209).
Improve model class hierarchy for robots. Now, robots own a subset of models (gripper(s), mount(s), etc.), allowing easy external access to the robot's internal model hierarchy.
robosuite docs updated
Add new papers

Critical Bug Fixes

Fix OSC global orientation limits (Quick fix in indices #228)

Other Bug Fixes

Fix default OSC orientation control (valid default rotation matrix) (Fixed default control for orientation in OSC #232)
Fix Jaco self-collisions (Fix Jaco Collision Geoms #235)
Fix joint velocity controller clipping and tune default kp (Fix Joint Velocity Bug #236)

Contributor Spotlight

A big thank you to the following community members for spearheading the renderer PRs for this release!
@awesome-aj0123
@divyanshj16

Merge latest public HEAD to robosuite-dev

…to robosuite-visii-branch

…tiative/robosuite-dev into robosuite-visii-branch

update roboturk docs with links to robomimic

Robosuite visii branch doc

…tiative/robosuite-dev into robosuite-visii-branch

addition of NVisII and iGibson renderers

…to remove the need to install ig assets

amandlek

Read through some portions of the code as a sanity check - it's likely that some functionality will break with the default mujoco renderers (see my comments).

README.md

amandlek · 2021-10-19T05:11:52Z

robosuite/environments/base.py

@@ -58,38 +56,29 @@ def __new__(meta, name, bases, class_dict):
 class MujocoEnv(metaclass=EnvMeta):
    """
    Initializes a Mujoco Environment.
-
    Args:
        has_renderer (bool): If true, render the simulation state in


Are these renderer arguments ignored if not using the mujoco / default renderers? We need to update docstrings + documentation online if that's the case - it's an important detail.

amandlek · 2021-10-19T05:12:02Z

robosuite/environments/base.py

+        """
+        Gets the pixel observations for the environment from the specified renderer
+        """
+        self.viewer.get_pixel_obs()


This seems like it will break for the default / mujoco renderer. We should also probably make sure here that off-screen rendering is enabled (if using the mujoco renderer).

amandlek · 2021-10-19T05:13:00Z

robosuite/environments/base.py

@@ -484,16 +510,22 @@ def visualize(self, vis_settings):
        for obj in self.model.mujoco_objects:
            obj.set_sites_visibility(sim=self.sim, visible=vis_settings["env"])

+    def set_camera_pos_quat(self, camera_pos, camera_quat):


It seems odd to only offer this method for nvisii renderer and not others - any particular reason why (a) we need this method for the nvisii renderer or (b) we do not support it for other renderers?

I added this for iG renderer.

amandlek · 2021-10-19T05:13:29Z

robosuite/environments/base.py

        Args:
            xml_string (str): Filepath to the xml file that will be loaded directly into the sim
        """

        # if there is an active viewer window, destroy it
-        self.close()
+        if self.renderer != 'nvisii':


why is this not done for the nvisii renderer? more comments would help

amandlek · 2021-10-19T05:14:25Z

robosuite/environments/robot_env.py

@@ -10,6 +10,11 @@
 from robosuite.robots import ROBOT_CLASS_MAPPING
 from robosuite.controllers import reset_controllers

+try:


why is this snippet here? we should be able to delete it right?

yes we can delete this, it was just left when I was cleaning up.

Deleted these lines.

robosuite/scripts/playback_demonstrations_from_hdf5.py

amandlek · 2021-10-19T05:19:58Z

robosuite/renderers/igibson/igibson_utils.py

+
+try:
+    import torch
+    HAS_TORCH = True


are we still using these flags? I thought we agreed on an alternative, or at least setting a Macro (utils/macros.py)

This flag will only be used when using the iG renderer. If HAS_TORCH is False and someone tries to use render2tensor it will raise an Exception.

added cube and sphere to the robosuite assets and modified ig bridge …

added reqs for iG renderer

Address remaining issues in #260 * remove unnecessary code and HAS_TORCH flag * add set_camera_pos_quat function for iG

abhihjoshi and others added 30 commits January 5, 2021 10:37

Base visii wrapper

b69993d

Updated robosuite-dev master to public HEAD

a072d63

Merge pull request #3 from ARISE-Initiative/public_head

aeb87d9

Merge latest public HEAD to robosuite-dev

Adding 4K textures

d55736a

Update visii_wrapper.py

3f27b96

splitting visii wrapper into multiple files

0a34c2e

Delete visii_wrapper.py

e982fb6

Create visii_rander_wrapper.py

bed5a99

Rename visii_rander_wrapper.py to visii_render_wrapper.py

35f714b

adding visii rendering files

78321ce

adding .obj and .mtl files for some objects

8375572

Delete visii_render_wrapper.py

1a6a952

Delete visii_utils.py

c5bed6f

Delete rendering_objects.py

2e8017b

Delete dynamic_object_initialization.py

283dcd3

Delete static_object_initialization.py

c1c1150

Merge remote-tracking branch 'upstream/master'

190f93e

Update requirements-extra.txt

f0d36d5

update for nvisii rendering

9883341

Merge branch 'master' of github.com:ARISE-Initiative/robosuite-dev in…

758f487

…to robosuite-visii-branch

add assets

bc5e4c5

added UV maps

5eaa148

added uv maps for baxter

a38f6c2

add igibson renderer

49ae646

iGWrapper bugs resolved

acebd09

fixes integration bugs in iG renderer and robosuite

a67182a

undo the change where camera_obs=True could be ON with has_renderer=True

4d11f00

fix material loading of objects like can, milk etc

e647869

adding demo script and whl file

dc52a6f

Merge branch 'robosuite-visii-branch' of https://github.com/ARISE-Ini…

b3fefe4

…tiative/robosuite-dev into robosuite-visii-branch

yukezhu and others added 22 commits October 4, 2021 15:24

Merge pull request #12 from ARISE-Initiative/update-roboturk-docs

dcfae7a

update roboturk docs with links to robomimic

fixing class/instance segmentations

e58e33a

add render FPS in documents

1dc92cc

Update renderers.md

04ad008

Merge pull request #13 from ARISE-Initiative/robosuite-visii-branch-doc

6fb46f1

Robosuite visii branch doc

update nvisii rendering image

d6a9550

reverting to old mujoco env

32bf698

Merge branch 'robosuite-visii-branch' of https://github.com/ARISE-Ini…

03091c6

…tiative/robosuite-dev into robosuite-visii-branch

adding mujoco on screen rendering fix

a0510b1

adding mujoco on screen rendering fix

a6c3488

adding observables fix

aa86a01

adding observables fix

afbd495

adding observables fix

6e0f68a

add one more profiling table

aeb4f55

Update renderers.md

d5b8a54

fixing demo renderers script

50fddba

Merge branch 'robosuite-visii-branch' of https://github.com/ARISE-Ini…

25e251d

…tiative/robosuite-dev into robosuite-visii-branch

fixing env base

ff7f354

Merge branch 'robosuite-visii-branch' of https://github.com/ARISE-Ini…

4e4b217

…tiative/robosuite-dev into robosuite-visii-branch

Merge pull request #4 from ARISE-Initiative/robosuite-visii-branch

157a8b4

addition of NVisII and iGibson renderers

Merge branch 'master' into v1.3

651b8dc

added cube and sphere to the robosuite assets and modified ig bridge …

1f9a6bf

…to remove the need to install ig assets

amandlek requested changes Oct 19, 2021

View reviewed changes

Roberto Martin-Martin and others added 5 commits October 18, 2021 22:27

added reqs for iG renderer

32c7957

Merge pull request #14 from ARISE-Initiative/removing_dependencies

389b54a

added cube and sphere to the robosuite assets and modified ig bridge …

Merge pull request #15 from ARISE-Initiative/docs_ig_reqs

65e175d

added reqs for iG renderer

Merge branch 'v1.3' of github.com:ARISE-Initiative/robosuite into v1.3

15209ed

incorporate comments from Ajay

37ea481

yukezhu merged commit c7d0b51 into master Oct 19, 2021

yukezhu added a commit that referenced this pull request Nov 3, 2021

v1.3 updates (#265)

a08b5a9

Address remaining issues in #260 * remove unnecessary code and HAS_TORCH flag * add set_camera_pos_quat function for iG

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

robosuite v1.3 release #260

robosuite v1.3 release #260

yukezhu commented Oct 19, 2021 •

edited by amandlek

Loading

amandlek left a comment

amandlek Oct 19, 2021

amandlek Oct 19, 2021

amandlek Oct 19, 2021

divyanshj16 Oct 25, 2021

amandlek Oct 19, 2021

amandlek Oct 19, 2021

divyanshj16 Oct 19, 2021

divyanshj16 Oct 25, 2021

amandlek Oct 19, 2021

divyanshj16 Oct 19, 2021

robosuite v1.3 release #260

robosuite v1.3 release #260

Conversation

yukezhu commented Oct 19, 2021 • edited by amandlek Loading

robosuite 1.3.0 Release Notes

Highlights

Renderers

Vision Modalities

Camera Utilities

Improvements

Critical Bug Fixes

Other Bug Fixes

Contributor Spotlight

amandlek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yukezhu commented Oct 19, 2021 •

edited by amandlek

Loading