ARISE-Initiative · yukezhu · Oct 19, 2021 · Jan 5, 2021 · Jan 24, 2021 · Jan 25, 2021
diff --git a/AUTHORS b/AUTHORS
@@ -24,4 +24,6 @@ Rachel Gardner <[email protected]>
 Jonathan Booher <[email protected]>
 Danfei Xu <[email protected]>
 Rachel Gardner <[email protected]>
-Albert Tung <[email protected]>
+Albert Tung <[email protected]>
+Abhishek Joshi <[email protected]>
+Divyansh Jha <[email protected]>
diff --git a/LICENSE b/LICENSE
@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2020 Stanford Vision and Learning Lab and UT-Austin Robot Perception and Learning Lab
+Copyright (c) 2021 Stanford Vision and Learning Lab and UT Robot Perception and Learning Lab
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
@@ -18,4 +18,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+SOFTWARE.
diff --git a/README.md b/README.md
@@ -6,30 +6,32 @@
 
 -------
 ## Latest Updates
-[02/17/2021] **v1.2.0**: Added observable sensor models :eyes: and dynamics randomization :game_die:
+[09/20/2021] **v1.3**: Ray tracing and physically based rendering tools :sparkles: and access to vision modalities 🎥
 
-[12/17/2020] **v1.1.0**: Refactored infrastructure and standardized model classes for much easier environment prototyping :wrench:
+[02/17/2021] **v1.2**: Added observable sensor models :eyes: and dynamics randomization :game_die:
+
+[12/17/2020] **v1.1**: Refactored infrastructure and standardized model classes for much easier environment prototyping :wrench:
 
 -------
 
-**robosuite** is a simulation framework powered by the [MuJoCo](http://mujoco.org/) physics engine for robot learning. It also offers a suite of benchmark environments for reproducible research. The current release (v1.2) features manipulation tasks with feature supports of procedural generation, advanced controllers, teleoperation, etc. This project is part of the broader [Advancing Robot Intelligence through Simulated Environments (ARISE) Initiative](https://github.com/ARISE-Initiative), with the aim of lowering the barriers of entry for cutting-edge research at the intersection of AI and Robotics.
+**robosuite** is a simulation framework powered by the [MuJoCo](http://mujoco.org/) physics engine for robot learning. It also offers a suite of benchmark environments for reproducible research. The current release (v1.3) features rendering tools, ground-truth of vision modalities, and camera utilities. This project is part of the broader [Advancing Robot Intelligence through Simulated Environments (ARISE) Initiative](https://github.com/ARISE-Initiative), with the aim of lowering the barriers of entry for cutting-edge research at the intersection of AI and Robotics.
 
 Data-driven algorithms, such as reinforcement learning and imitation learning, provide a powerful and generic tool in robotics. These learning paradigms, fueled by new advances in deep learning, have achieved some exciting successes in a variety of robot control problems. However, the challenges of reproducibility and the limited accessibility of robot hardware (especially during a pandemic) have impaired research progress. The overarching goal of **robosuite** is to provide researchers with:
 
-* a standardized set of benchmarking tasks for rigorus evaluation and algorithm development;
+* a standardized set of benchmarking tasks for rigorous evaluation and algorithm development;
 * a modular design that offers great flexibility to design new robot simulation environments;
 * a high-quality implementation of robot controllers and off-the-shelf learning algorithms to lower the barriers to entry.
 
-This framework was originally developed since late 2017 by researchers in [Stanford Vision and Learning Lab](http://svl.stanford.edu) (SVL) as an internal tool for robot learning research. Now it is actively maintained and used for robotics research projects in SVL and the [UT-Austin Robot Perception and Learning Lab](http://rpl.cs.utexas.edu) (RPL). We welcome community contributions to this project. For details please check out our [contributing guidelines](CONTRIBUTING.md).
+This framework was originally developed since late 2017 by researchers in [Stanford Vision and Learning Lab](http://svl.stanford.edu) (SVL) as an internal tool for robot learning research. Now it is actively maintained and used for robotics research projects in SVL and the [UT Robot Perception and Learning Lab](http://rpl.cs.utexas.edu) (RPL). We welcome community contributions to this project. For details please check out our [contributing guidelines](CONTRIBUTING.md).
 
 This release of **robosuite** contains seven robot models, eight gripper models, six controller modes, and nine standardized tasks. It also offers a modular design of APIs for building new environments with procedural generation. We highlight these primary features below:
 
 * **standardized tasks**: a set of standardized manipulation tasks of large diversity and varying complexity and RL benchmarking results for reproducible research;
-* **procedural generation**: modular APIs for programmatically creating new environments and new tasks as a combinations of robot models, arenas, and parameterized 3D objects;
+* **procedural generation**: modular APIs for programmatically creating new environments and new tasks as combinations of robot models, arenas, and parameterized 3D objects;
 * **controller supports**: a selection of controller types to command the robots, such as joint-space velocity control, inverse kinematics control, operational space control, and 3D motion devices for teleoperation;
 * **multi-modal sensors**: heterogeneous types of sensory signals, including low-level physical states, RGB cameras, depth maps, and proprioception;
-* **human demonstrations**: utilities for collecting human demonstrations, replaying demonstration datasets, and leveraging demonstration data for learning.
-
+* **human demonstrations**: utilities for collecting human demonstrations, replaying demonstration datasets, and leveraging demonstration data for learning. Check out our sister project [robomimic](https://arise-initiative.github.io/robomimic-web/);
+* **photorealistic rendering**: integration with advanced graphics tools that provide real-time photorealistic renderings of simulated scenes.
 
 ## Citations
 Please cite [**robosuite**](https://robosuite.ai) if you use this framework in your publications:

diff --git a/docs/acknowledgement.md b/docs/acknowledgement.md
@@ -1,12 +1,13 @@
 # Acknowledgements
 
-**robosuite** is built on the [MuJoCo engine](http://www.mujoco.org/) with the Python interfaces provided by [mujoco-py](https://github.com/openai/mujoco-py). We would like to thank members of the [Stanford People, AI, & Robots (PAIR) Group](http://pair.stanford.edu/) for their support and feedback to this project. In particular, the following people have made their contributions in different stages of this project:
+**robosuite** is built on the [MuJoCo engine](http://www.mujoco.org/) with the Python interfaces provided by [mujoco-py](https://github.com/openai/mujoco-py). We would like to thank members of the [Stanford People, AI, & Robots (PAIR) Group](http://pair.stanford.edu/) and [UT Robot Perception and Learning Lab](http://rpl.cs.utexas.edu/) for their support and feedback to this project. In particular, the following people have made their contributions in different stages of this project:
 
 - [Jiren Zhu](https://github.com/jirenz), [Joan Creus-Costa](https://github.com/jcreus) (robosuite v0.3)
 - [Jim (Linxi) Fan](http://jimfan.me/), [Zihua Liu](https://www.linkedin.com/in/zihua-liu/), [Orien Zeng](https://www.linkedin.com/in/orien-zeng-054589b6/), [Anchit Gupta](https://www.linkedin.com/in/anchitgupta/) ([Surreal](http://surreal.stanford.edu/) experiments)
 - [Michelle Lee](http://stanford.edu/~mishlee/), [Rachel Gardner](https://www.linkedin.com/in/rachel0/) (controller implementations)
 - [Danfei Xu](https://cs.stanford.edu/~danfei/) (placement initializer)
 - [Andrew Kondrich](http://www.andrewkondrich.com/), [Jonathan Booher](https://web.stanford.edu/~jaustinb/) (domain randomization)
 - [Albert Tung](https://www.linkedin.com/in/albert-tung3/) (demonstration collection)
+- [Abhishek Joshi](https://www.linkedin.com/in/abhishek-joshi-4ab469180), [Divyansh Jha](https://github.com/divyanshj16) (robosuite v1.3 renderers)
 
 We wholeheartedly welcome the community to contribute to our project through issues and pull requests. New contributors will be added to the list above.
diff --git a/docs/algorithms/demonstrations.md b/docs/algorithms/demonstrations.md
@@ -45,7 +45,7 @@ We have included an example script that illustrates how demonstrations can be lo
 
 We have included some sample demonstrations for each task at `models/assets/demonstrations`.
 
-Our twin project [RoboTurk](http://roboturk.stanford.edu) has also collected pilot datasets of more than a thousand demonstrations for two tasks in our suite via crowdsourcing. You can find detailed information about the RoboTurk datasets [here](roboturk).
+Our sister project [RoboTurk](http://roboturk.stanford.edu) has also collected several human demonstration datasets across different tasks and humans, including pilot datasets of more than a thousand demonstrations for two tasks in our suite via crowdsourcing. You can find detailed information about the RoboTurk datasets [here](roboturk).
 
 
 ## Structure of collected demonstrations
@@ -81,7 +81,9 @@ The reason for storing mujoco states instead of raw observations is to make it e
 
 ## Using Demonstrations for Learning
 
-[Several](https://arxiv.org/abs/1802.09564) [prior](https://arxiv.org/abs/1807.06919) [works](https://arxiv.org/abs/1804.02717) have demonstrated the effectiveness of altering the start state distribution of training episodes for learning RL policies. We provide a generic utility for setting various types of learning curriculums which dictate how to sample from demonstration episodes when doing an environment reset. For more information see the `DemoSamplerWrapper` class.
+We have recently released the [robomimic](https://arise-initiative.github.io/robomimic-web/) framework, which makes it easy to train policies using your own [datasets collected with robosuite](https://arise-initiative.github.io/robomimic-web/docs/introduction/datasets.html#robosuite-hdf5-datasets), and other publically released datasets (such as those collected with RoboTurk). The framework also contains many useful examples for how to integrate hdf5 datasets into your own learning pipeline.
+
+The robosuite repository also has some utilities for using the demonstrations to alter the start state distribution of training episodes for learning RL policies - this have proved effective in [several](https://arxiv.org/abs/1802.09564) [prior](https://arxiv.org/abs/1807.06919) [works](https://arxiv.org/abs/1804.02717). For example, we provide a generic utility for setting various types of learning curriculums which dictate how to sample from demonstration episodes when doing an environment reset. For more information see the `DemoSamplerWrapper` class.
 
 ## Warnings
 We have verified that deterministic action playback works specifically when playing back demonstrations on the *same machine* that the demonstrations were originally collected upon. However, this means that deterministic action playback is NOT guaranteed (in fact, very unlikely) to work across platforms or even across different machines using the same OS.

diff --git a/docs/algorithms/roboturk.md b/docs/algorithms/roboturk.md
@@ -2,18 +2,20 @@
 
 [RoboTurk](https://roboturk.stanford.edu/) is a crowdsourcing platform developed in order to enabled collecting large-scale manipulation datasets. Below, we describe RoboTurk datasets that are compatible with robosuite.
 
-## Updated Datasets compatible with v1.0+
+## Datasets compatible with v1.2+
 
-We are currently in the process of organizing a standardized dataset for our benchmarking tasks, which will be made available soon and compatible with v1.2.0+. In the meantime, we have provided a [small-scale dataset](https://drive.google.com/drive/folders/1LLkuFnRdqQ6xn1cYzkbJUs_DreaAvN7i?usp=sharing) of expert demonstrations on two of our tasks.
+We have collected several human demonstration datasets across several tasks implemented in robosuite as part of the [robomimic](https://arise-initiative.github.io/robomimic-web/) framework. For more information on these datasets, including how to download them and start training policies with them, please see [this link](https://arise-initiative.github.io/robomimic-web/docs/introduction/results.html#downloading-released-datasets).
 
-## Original Datasets compatible with v0.3
+## Datasets compatible with v0.3
 
 We collected a large-scale dataset on the `SawyerPickPlace` and `SawyerNutAssembly` tasks using the [RoboTurk](https://crowdncloud.ai/) platform. Crowdsourced workers collected these task demonstrations remotely. It consists of **1070** successful `SawyerPickPlace` demonstrations and **1147** successful `SawyerNutAssembly` demonstrations.
 
 We are providing the dataset in the hopes that it will be beneficial to researchers working on imitation learning. Large-scale imitation learning has not been explored much in the community; it will be exciting to see how this data is used.
 
 You can download the dataset [here](http://cvgl.stanford.edu/projects/roboturk/RoboTurkPilot.zip).
 
+**Note:** to get started with this data, we highly recommend using the [robomimic](https://arise-initiative.github.io/robomimic-web/) framework - see [this link](https://arise-initiative.github.io/robomimic-web/docs/introduction/datasets.html#roboturk-pilot-datasets) for more information. To use this data, you should be on the [roboturk_v1](https://github.com/ARISE-Initiative/robosuite/tree/roboturk_v1) branch of robosuite, which is `v0.3` with a few minor changes. You can do this by using `git checkout roboturk_v1` after cloning the repository, or just download the source code from [this link](https://github.com/ARISE-Initiative/robosuite/tree/roboturk_v1).
+
 After unzipping the dataset, the following subdirectories can be found within the `RoboTurkPilot` directory.
 
 - **bins-full**

diff --git a/docs/conf.py b/docs/conf.py
@@ -65,7 +65,7 @@
 
 # General information about the project.
 project = u'robosuite'
-copyright = u'Stanford University and The University of Texas at Austin 2020'
+copyright = u'Stanford University and The University of Texas at Austin 2021'
 author = u'Yuke Zhu, Josiah Wong, Jiren Zhu, Ajay Mandlekar, Roberto Martín-Martín'
 
 # The version info for the project you're documenting, acts as replacement for

diff --git a/docs/demos.md b/docs/demos.md
@@ -182,3 +182,20 @@ The `demo_video_recording.py` script shows how to record a video of robot roll-o
 ```sh
 $ python demo_video_recording.py --environment Lift --robots Panda
 ```
+
+### Rendering Options
+The `demo_renderers.py` script shows how to use different renderers with the simulation environments. Our current version supports three rendering options: MuJoCo (default), NVISII, and iGibson renderers. More information about these renderers can be found in the [Renderer](modules/renderers) module. Example:
+```sh
+$ python demo_renderers.py --renderer igibson
+```
+The `--renderer` flag can be set to `mujoco` (default), `nvisii` and `igibson`.
+
+### Vision Modalities
+The `demo_igibson_modalities.py` and `demo_nvisii_modalities.py` scripts illustrate how to obtain vision modalities from the iGibson renderer and NVISII renderer respectively. This script uses the flags specified and renders that particular vision modality. Example:
+```sh
+$ python demo_igibson_modalities.py --vision-modality segmentation --segmentation-level instance
+
+$ python demo_nvisii_modalities.py --vision-modality depth
+```
+The `--vision-modality` flag can be set to `depth`, `normal`, `segmentation` or `rgb` (default).
+The `-segmentation-level` flag can be set only when `--vision-modality` is set to `segmentation`. It can set to `instance`, `class`, or `element`.
diff --git a/docs/images/renderers/renderers.png b/docs/images/renderers/renderers.png
diff --git a/docs/images/renderers/vision_modalities_igibson.png b/docs/images/renderers/vision_modalities_igibson.png
diff --git a/docs/images/renderers/vision_modalities_nvisii.png b/docs/images/renderers/vision_modalities_nvisii.png