robosuite v1.3 release (#260)

# robosuite 1.3.0 Release Notes
- Highlights
- New Features
- Improvements
- Critical Bug Fixes
- Other Bug Fixes

# Highlights
This release of robosuite brings powerful rendering functionalities including new renderers and multiple vision modalities, in addition to some general-purpose camera utilities. Below, we discuss the key details of these new features:

## Renderers
In addition to the native Mujoco renderer, we present two new renderers, [NVISII](https://github.com/owl-project/NVISII) and [iGibson](http://svl.stanford.edu/igibson/), and introduce a standardized rendering interface API to enable easy swapping of renderers.

NVISII is a high-fidelity ray-tracing renderer originally developed by NVIDIA, and adapted for plug-and-play usage in **robosuite**. It is primarily used for training perception models and visualizing results in high quality. It can run at up to ~0.5 fps using a GTX 1080Ti GPU. Note that NVISII must be installed (`pip install nvisii`) in order to use this renderer.

iGibson is a much faster renderer that additionally supports physics-based rendering (PBR) and direct rendering to pytorch tensors. While not as high-fidelity as NVISII, it is incredibly fast and can run at up to ~1500 fps using a GTX 1080Ti GPU. Note that iGibson must be installed (`pip install igibson`) in order to use this renderer.

With the addition of these new renderers, we also introduce a standardized [renderer](https://github.com/ARISE-Initiative/robosuite/blob/master/robosuite/renderers/base.py) for easy usage and customization of the various renderers. During each environment step, the renderer updates its internal state by calling `update()` and renders by calling `render(...)`. The resulting visual observations can be polled by calling `get_pixel_obs()` or by calling other methods specific to individual renderers. We provide a [demo script](https://github.com/ARISE-Initiative/robosuite/blob/master/robosuite/demos/demo_segmentation.py) for testing each new renderer, and our docs also provide [additional information](http://robosuite.ai/docs/modules/renderers.md) on specific renderer details and installation procedures.

## Vision Modalities
In addition to new renderers, we also provide broad support for multiple vision modalities across all (Mujoco, NVISII, iGibson) renderers:

- **RGB**: Standard 3-channel color frames with values in range `[0, 255]`. This is set during environment construction with the `use_camera_obs` argument.
- **Depth**: 1-channel frame with normalized values in range `[0, 1]`. This is set during environment construction with the `camera_depths` argument.
- **Segmentation**: 1-channel frames with pixel values corresponding to integer IDs for various objects. Segmentation can occur by class, instance, or geom, and is set during environment construction with the `camera_segmentations` argument.
    
In addition to the above modalities, the following modalities are supported by a subset of renderers:

- **Surface Normals**: [NVISII, iGibson] 3-channel (x,y,z) normalized direction vectors.
- **Texture Coordinates**: [NVISII] 3-channel (x,y,z) coordinate texture mappings for each element
- **Texture Positioning**: [NVISII, iGibson] 3-channel (x,y,z) global coordinates of each pixel.

Specific modalities can be set during environment and renderer construction. We provide a [demo script](https://github.com/ARISE-Initiative/robosuite/blob/master/robosuite/demos/demo_nvisii_modalities.py) for testing the different modalities supported by NVISII and a [demo script](https://github.com/ARISE-Initiative/robosuite/blob/master/robosuite/demos/demo_igibson_modalities.py) for testing the different modalities supported by iGibson.

## Camera Utilities
We provide a set of general-purpose [camera utilities](https://github.com/ARISE-Initiative/robosuite/blob/master/robosuite/utils/camera_utils.py) that intended to enable easy manipulation of environment cameras. Of note, we include transform utilities for mapping between pixel, camera, and world frames, and include a [CameraMover](https://github.com/ARISE-Initiative/robosuite/blob/master/robosuite/utils/camera_utils.py#L244) class for dynamically moving a camera during simulation, which can be used for many purposes such as the [DemoPlaybackCameraMover](https://github.com/ARISE-Initiative/robosuite/blob/master/robosuite/utils/camera_utils.py#L419) subclass that enables smooth visualization during demonstration playback.

# Improvements
The following briefly describes other changes that improve on the pre-existing structure. This is not an exhaustive list, but a highlighted list of changes.

- Standardize EEF frames (#204). Now, all grippers have identical conventions for plug-and-play usage across types.

- Add OSC_POSITION control option for spacemouse (#209).

- Improve model class hierarchy for robots. Now, robots own a subset of models (gripper(s), mount(s), etc.), allowing easy external access to the robot's internal model hierarchy.

- robosuite docs updated

- Add new papers


# Critical Bug Fixes
- Fix OSC global orientation limits (#228)


# Other Bug Fixes
- Fix default OSC orientation control (valid default rotation matrix) (#232)

- Fix Jaco self-collisions (#235)

- Fix joint velocity controller clipping and tune default kp (#236)

-------

## Contributor Spotlight
A big thank you to the following community members for spearheading the renderer PRs for this release!
@awesome-aj0123 
@divyanshj16

AUTHORS

@@ -24,4 +24,6 @@ Rachel Gardner <[email protected]>
 Jonathan Booher <[email protected]>
 Danfei Xu <[email protected]>
 Rachel Gardner <[email protected]>
-Albert Tung <[email protected]>
+Albert Tung <[email protected]>
+Abhishek Joshi <[email protected]>
+Divyansh Jha <[email protected]>

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2020 Stanford Vision and Learning Lab and UT-Austin Robot Perception and Learning Lab
+Copyright (c) 2021 Stanford Vision and Learning Lab and UT Robot Perception and Learning Lab
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
@@ -18,4 +18,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+SOFTWARE.

README.md

@@ -6,30 +6,32 @@
 
 -------
 ## Latest Updates
-[02/17/2021] **v1.2.0**: Added observable sensor models :eyes: and dynamics randomization :game_die:
+[10/19/2021] **v1.3**: Ray tracing and physically based rendering tools :sparkles: and access to additional vision modalities 🎥
 
-[12/17/2020] **v1.1.0**: Refactored infrastructure and standardized model classes for much easier environment prototyping :wrench:
+[02/17/2021] **v1.2**: Added observable sensor models :eyes: and dynamics randomization :game_die:
+
+[12/17/2020] **v1.1**: Refactored infrastructure and standardized model classes for much easier environment prototyping :wrench:
 
 -------
 
-**robosuite** is a simulation framework powered by the [MuJoCo](http://mujoco.org/) physics engine for robot learning. It also offers a suite of benchmark environments for reproducible research. The current release (v1.2) features manipulation tasks with feature supports of procedural generation, advanced controllers, teleoperation, etc. This project is part of the broader [Advancing Robot Intelligence through Simulated Environments (ARISE) Initiative](https://github.com/ARISE-Initiative), with the aim of lowering the barriers of entry for cutting-edge research at the intersection of AI and Robotics.
+**robosuite** is a simulation framework powered by the [MuJoCo](http://mujoco.org/) physics engine for robot learning. It also offers a suite of benchmark environments for reproducible research. The current release (v1.3) features rendering tools, ground-truth of vision modalities, and camera utilities. This project is part of the broader [Advancing Robot Intelligence through Simulated Environments (ARISE) Initiative](https://github.com/ARISE-Initiative), with the aim of lowering the barriers of entry for cutting-edge research at the intersection of AI and Robotics.
 
 Data-driven algorithms, such as reinforcement learning and imitation learning, provide a powerful and generic tool in robotics. These learning paradigms, fueled by new advances in deep learning, have achieved some exciting successes in a variety of robot control problems. However, the challenges of reproducibility and the limited accessibility of robot hardware (especially during a pandemic) have impaired research progress. The overarching goal of **robosuite** is to provide researchers with:
 
-* a standardized set of benchmarking tasks for rigorus evaluation and algorithm development;
+* a standardized set of benchmarking tasks for rigorous evaluation and algorithm development;
 * a modular design that offers great flexibility to design new robot simulation environments;
 * a high-quality implementation of robot controllers and off-the-shelf learning algorithms to lower the barriers to entry.
 
-This framework was originally developed since late 2017 by researchers in [Stanford Vision and Learning Lab](http://svl.stanford.edu) (SVL) as an internal tool for robot learning research. Now it is actively maintained and used for robotics research projects in SVL and the [UT-Austin Robot Perception and Learning Lab](http://rpl.cs.utexas.edu) (RPL). We welcome community contributions to this project. For details please check out our [contributing guidelines](CONTRIBUTING.md).
+This framework was originally developed since late 2017 by researchers in [Stanford Vision and Learning Lab](http://svl.stanford.edu) (SVL) as an internal tool for robot learning research. Now it is actively maintained and used for robotics research projects in SVL and the [UT Robot Perception and Learning Lab](http://rpl.cs.utexas.edu) (RPL). We welcome community contributions to this project. For details please check out our [contributing guidelines](CONTRIBUTING.md).
 
 This release of **robosuite** contains seven robot models, eight gripper models, six controller modes, and nine standardized tasks. It also offers a modular design of APIs for building new environments with procedural generation. We highlight these primary features below:
 
 * **standardized tasks**: a set of standardized manipulation tasks of large diversity and varying complexity and RL benchmarking results for reproducible research;
-* **procedural generation**: modular APIs for programmatically creating new environments and new tasks as a combinations of robot models, arenas, and parameterized 3D objects;
-* **controller supports**: a selection of controller types to command the robots, such as joint-space velocity control, inverse kinematics control, operational space control, and 3D motion devices for teleoperation;
+* **procedural generation**: modular APIs for programmatically creating new environments and new tasks as combinations of robot models, arenas, and parameterized 3D objects;
+* **robot controllers**: a selection of controller types to command the robots, such as joint-space velocity control, inverse kinematics control, operational space control, and 3D motion devices for teleoperation;
 * **multi-modal sensors**: heterogeneous types of sensory signals, including low-level physical states, RGB cameras, depth maps, and proprioception;
-* **human demonstrations**: utilities for collecting human demonstrations, replaying demonstration datasets, and leveraging demonstration data for learning.
-
+* **human demonstrations**: utilities for collecting human demonstrations, replaying demonstration datasets, and leveraging demonstration data for learning. Check out our sister project [robomimic](https://arise-initiative.github.io/robomimic-web/);
+* **photorealistic rendering**: integration with advanced graphics tools that provide real-time photorealistic renderings of simulated scenes.
 
 ## Citations
 Please cite [**robosuite**](https://robosuite.ai) if you use this framework in your publications:

docs/acknowledgement.md

@@ -1,12 +1,13 @@
 # Acknowledgements
 
-**robosuite** is built on the [MuJoCo engine](http://www.mujoco.org/) with the Python interfaces provided by [mujoco-py](https://github.com/openai/mujoco-py). We would like to thank members of the [Stanford People, AI, & Robots (PAIR) Group](http://pair.stanford.edu/) for their support and feedback to this project. In particular, the following people have made their contributions in different stages of this project:
+**robosuite** is built on the [MuJoCo engine](http://www.mujoco.org/) with the Python interfaces provided by [mujoco-py](https://github.com/openai/mujoco-py). We would like to thank members of the [Stanford People, AI, & Robots (PAIR) Group](http://pair.stanford.edu/) and [UT Robot Perception and Learning Lab](http://rpl.cs.utexas.edu/) for their support and feedback to this project. In particular, the following people have made their contributions in different stages of this project:
 
 - [Jiren Zhu](https://github.com/jirenz), [Joan Creus-Costa](https://github.com/jcreus) (robosuite v0.3)
 - [Jim (Linxi) Fan](http://jimfan.me/), [Zihua Liu](https://www.linkedin.com/in/zihua-liu/), [Orien Zeng](https://www.linkedin.com/in/orien-zeng-054589b6/), [Anchit Gupta](https://www.linkedin.com/in/anchitgupta/) ([Surreal](http://surreal.stanford.edu/) experiments)
 - [Michelle Lee](http://stanford.edu/~mishlee/), [Rachel Gardner](https://www.linkedin.com/in/rachel0/) (controller implementations)
 - [Danfei Xu](https://cs.stanford.edu/~danfei/) (placement initializer)
 - [Andrew Kondrich](http://www.andrewkondrich.com/), [Jonathan Booher](https://web.stanford.edu/~jaustinb/) (domain randomization)
 - [Albert Tung](https://www.linkedin.com/in/albert-tung3/) (demonstration collection)
+- [Abhishek Joshi](https://www.linkedin.com/in/abhishek-joshi-4ab469180), [Divyansh Jha](https://github.com/divyanshj16) (robosuite v1.3 renderers)
 
 We wholeheartedly welcome the community to contribute to our project through issues and pull requests. New contributors will be added to the list above.

docs/algorithms/demonstrations.md

@@ -45,7 +45,7 @@ We have included an example script that illustrates how demonstrations can be lo
 
 We have included some sample demonstrations for each task at `models/assets/demonstrations`.
 
-Our twin project [RoboTurk](http://roboturk.stanford.edu) has also collected pilot datasets of more than a thousand demonstrations for two tasks in our suite via crowdsourcing. You can find detailed information about the RoboTurk datasets [here](roboturk).
+Our sister project [RoboTurk](http://roboturk.stanford.edu) has also collected several human demonstration datasets across different tasks and humans, including pilot datasets of more than a thousand demonstrations for two tasks in our suite via crowdsourcing. You can find detailed information about the RoboTurk datasets [here](roboturk).
 
 
 ## Structure of collected demonstrations
@@ -81,7 +81,9 @@ The reason for storing mujoco states instead of raw observations is to make it e
 
 ## Using Demonstrations for Learning
 
-[Several](https://arxiv.org/abs/1802.09564) [prior](https://arxiv.org/abs/1807.06919) [works](https://arxiv.org/abs/1804.02717) have demonstrated the effectiveness of altering the start state distribution of training episodes for learning RL policies. We provide a generic utility for setting various types of learning curriculums which dictate how to sample from demonstration episodes when doing an environment reset. For more information see the `DemoSamplerWrapper` class.
+We have recently released the [robomimic](https://arise-initiative.github.io/robomimic-web/) framework, which makes it easy to train policies using your own [datasets collected with robosuite](https://arise-initiative.github.io/robomimic-web/docs/introduction/datasets.html#robosuite-hdf5-datasets), and other publically released datasets (such as those collected with RoboTurk). The framework also contains many useful examples for how to integrate hdf5 datasets into your own learning pipeline.
+
+The robosuite repository also has some utilities for using the demonstrations to alter the start state distribution of training episodes for learning RL policies - this have proved effective in [several](https://arxiv.org/abs/1802.09564) [prior](https://arxiv.org/abs/1807.06919) [works](https://arxiv.org/abs/1804.02717). For example, we provide a generic utility for setting various types of learning curriculums which dictate how to sample from demonstration episodes when doing an environment reset. For more information see the `DemoSamplerWrapper` class.
 
 ## Warnings
 We have verified that deterministic action playback works specifically when playing back demonstrations on the *same machine* that the demonstrations were originally collected upon. However, this means that deterministic action playback is NOT guaranteed (in fact, very unlikely) to work across platforms or even across different machines using the same OS.

docs/algorithms/roboturk.md

@@ -2,18 +2,20 @@
 
 [RoboTurk](https://roboturk.stanford.edu/) is a crowdsourcing platform developed in order to enabled collecting large-scale manipulation datasets. Below, we describe RoboTurk datasets that are compatible with robosuite.
 
-## Updated Datasets compatible with v1.0+
+## Datasets compatible with v1.2+
 
-We are currently in the process of organizing a standardized dataset for our benchmarking tasks, which will be made available soon and compatible with v1.2.0+. In the meantime, we have provided a [small-scale dataset](https://drive.google.com/drive/folders/1LLkuFnRdqQ6xn1cYzkbJUs_DreaAvN7i?usp=sharing) of expert demonstrations on two of our tasks.
+We have collected several human demonstration datasets across several tasks implemented in robosuite as part of the [robomimic](https://arise-initiative.github.io/robomimic-web/) framework. For more information on these datasets, including how to download them and start training policies with them, please see [this link](https://arise-initiative.github.io/robomimic-web/docs/introduction/results.html#downloading-released-datasets).
 
-## Original Datasets compatible with v0.3
+## Datasets compatible with v0.3
 
 We collected a large-scale dataset on the `SawyerPickPlace` and `SawyerNutAssembly` tasks using the [RoboTurk](https://crowdncloud.ai/) platform. Crowdsourced workers collected these task demonstrations remotely. It consists of **1070** successful `SawyerPickPlace` demonstrations and **1147** successful `SawyerNutAssembly` demonstrations.
 
 We are providing the dataset in the hopes that it will be beneficial to researchers working on imitation learning. Large-scale imitation learning has not been explored much in the community; it will be exciting to see how this data is used.
 
 You can download the dataset [here](http://cvgl.stanford.edu/projects/roboturk/RoboTurkPilot.zip).
 
+**Note:** to get started with this data, we highly recommend using the [robomimic](https://arise-initiative.github.io/robomimic-web/) framework - see [this link](https://arise-initiative.github.io/robomimic-web/docs/introduction/datasets.html#roboturk-pilot-datasets) for more information. To use this data, you should be on the [roboturk_v1](https://github.com/ARISE-Initiative/robosuite/tree/roboturk_v1) branch of robosuite, which is `v0.3` with a few minor changes. You can do this by using `git checkout roboturk_v1` after cloning the repository, or just download the source code from [this link](https://github.com/ARISE-Initiative/robosuite/tree/roboturk_v1).
+
 After unzipping the dataset, the following subdirectories can be found within the `RoboTurkPilot` directory.
 
 - **bins-full**

docs/conf.py

@@ -65,7 +65,7 @@
 
 # General information about the project.
 project = u'robosuite'
-copyright = u'Stanford University and The University of Texas at Austin 2020'
+copyright = u'Stanford University and The University of Texas at Austin 2021'
 author = u'Yuke Zhu, Josiah Wong, Jiren Zhu, Ajay Mandlekar, Roberto Martín-Martín'
 
 # The version info for the project you're documenting, acts as replacement for

docs/demos.md

@@ -182,3 +182,20 @@ The `demo_video_recording.py` script shows how to record a video of robot roll-o
 ```sh
 $ python demo_video_recording.py --environment Lift --robots Panda
 ```
+
+### Rendering Options
+The `demo_renderers.py` script shows how to use different renderers with the simulation environments. Our current version supports three rendering options: MuJoCo (default), NVISII, and iGibson renderers. More information about these renderers can be found in the [Renderer](modules/renderers) module. Example:
+```sh
+$ python demo_renderers.py --renderer igibson
+```
+The `--renderer` flag can be set to `mujoco` (default), `nvisii` and `igibson`.
+
+### Vision Modalities
+The `demo_igibson_modalities.py` and `demo_nvisii_modalities.py` scripts illustrate how to obtain vision modalities from the iGibson renderer and NVISII renderer respectively. This script uses the flags specified and renders that particular vision modality. Example:
+```sh
+$ python demo_igibson_modalities.py --vision-modality segmentation --segmentation-level instance
+
+$ python demo_nvisii_modalities.py --vision-modality depth
+```
+The `--vision-modality` flag can be set to `depth`, `normal`, `segmentation` or `rgb` (default).
+The `-segmentation-level` flag can be set only when `--vision-modality` is set to `segmentation`. It can set to `instance`, `class`, or `element`.