SenseAct: A computational framework for real-world robot learning tasks

This repository provides the implementation of several reinforcement learning tasks with multiple real-world robots. These tasks come with an interface similar to OpenAI-Gym so that learning algorithms can be plugged in easily and in a uniform manner across tasks. All the tasks here are implemented based on a computational framework of robot-agent communication proposed by Mahmood et al. (2018a), which we call SenseAct. In this computational framework, agent and environment-related computations are ordered and distributed among multiple concurrent processes in a specific way. By doing so, SenseAct enables the following:

• Timely communication between the learning agent and multiple robotic devices with reduced latency,
• Easy and systematic design of robotic tasks for reinforcement learning agents,
• Facilitate reproducible real-world reinforcement learning.

This repository provides the following real-world robotic tasks, which are proposed by Mahmood et al. (2018b) as benchmark tasks for reinforcement learning algorithms:

Universal-Robots (UR) robotic arms:

Tested on UR Software v. 3.3.4.310

• UR-Reacher (both 2 joint and 6 joint control)

UR-Reacher-2	UR-Reacher-6

Dynamixel (DXL) actuators:

Currently we only support MX-64AT.

• DXL-Reacher
• DXL-Tracker

DXL-Reacher	DXL-Tracker

iRobot Create 2 robots:

• Create-Mover
• Create-Docker

Create-Mover	Create-Docker

Mahmood et al. (2018b) provide extensive results comparing multiple reinforcement learning algorithms on the above tasks, and Mahmood et al. (2018a) show the effect of different task-setup elements in learning. Their results can be reproduced by using this repository (see documentation for more information).

Versions

The branch master is the latest official release and dev is current development branch.

Installation

SenseAct uses Python3 (>=3.5), and all other requirements are automatically installed via pip.

On Linux and Mac OS X, run the following:

1. git clone https://github.com/kindredresearch/SenseAct.git
2. cd SenseAct
3. pip install -e . or pip3 install -e . depends on your setup

To replicate experimental results from the paper please install the tag v0.1.1 from the git repo.

1. git fetch --all --tags
2. git checkout tags/v0.1.1

Additional instruction for installing OpenAI Baselines needed for running the advanced examples is given in the corresponding readme.

Additional installation steps for Dynamixel-based tasks (Linux only)

Dynamixels can be controlled by drivers written using either ctypes by Robotis or pyserial, which can be chosen by passing either True (ctypes) or False (pyserial) as an argument to the use_ctypes_driver parameter of a Dynamixel-based task (e.g., see examples/advanced/dxl_reacher.py). We found the ctypes-based driver to provide substantially more timely and precise communication compared to the pyserial-based one.

In order to use the CType-based driver, we need to install gcc and relevant packages for compiling the C libraries:

sudo apt-get install gcc-5 build-essential gcc-multilib g++-multilib

Then run the following script to download and compile the Dynamixel driver C libraries:

sudo bash setup_dxl.sh

For additional setup and troubleshooting information regarding Dynamixels, please see DXL Docs.

Tests

You can check whether SenseAct is installed correctly by running the included unit tests.

cd SenseAct
python -m unittest discover -b

Support

Installation problems? Feature requests? General questions?

• read through examples and API documentation
• create github issues on the SenseAct project
• join the mailing list https://groups.google.com/forum/#!forum/senseact

Acknowledgments

This project is developed by the Kindred AI Research team. Rupam Mahmood, Dmytro Korenkevych, and Brent Komer originally developed the computational framework and the UR tasks. William Ma developed the Create 2 tasks and contributed substantially by adding new features to SenseAct. Gautham Vasan developed the DXL tasks. Francois Hogan developed the simulated task.

James Bergstra provided support and guidance throughout the development. Adrian Martin, Scott Rostrup, and Jonathan Yep developed the pyserial DXL driver for a Kindred project, which was used for the SenseAct DXL Communicator. Daniel Snider, Oliver Limoyo, Dylan Ashley, and Craig Sherstan tested the framework, provided thoughtful suggestions, and confirmed the reproducibility of learning by running experiments on real robots.

Citing SenseAct

For the SenseAct computational framework and the UR-Reacher tasks, please cite Mahmood et al. (2018a). For the DXL and the Create 2 tasks, please cite Mahmood et al. (2018b).

• Mahmood, A. R., Korenkevych, D., Komer,B. J., Bergstra, J. (2018a). Setting up a reinforcement learning task with a real-world robot. In IEEE/RSJ International Conference on Intelligent Robots and Systems.

• Mahmood, A. R., Korenkevych, D., Vasan, G., Ma, W., Bergstra, J. (2018b). Benchmarking reinforcement learning algorithms on real-world robots. In Proceedings of the 2nd Annual Conference on Robot Learning.