SaLinA: SaLinA - A Flexible and Simple Library for Learning Sequential Agents (including Reinforcement Learning)

Documentation: Read the docs

TL;DR.

salina is a lightweight library extending PyTorch modules for developping sequential decision models. It can be used for Reinforcement Learning (including model-based with differentiable environments, multi-agent RL, etc...), but also in a supervised/unsupervised learning settings (for instance for NLP, Computer Vision, etc...).

• It allows to write very complex sequential models (or policies) in few lines
• It works on multiple CPUs and GPUs

Citing salina

Please use this bibtex if you want to cite this repository in your publications:

Link to the paper: SaLinA: Sequential Learning of Agents

    @misc{salina,
        author = {Ludovic Denoyer, Alfredo de la Fuente, Song Duong, Jean-Baptiste Gaya, Pierre-Alexandre Kamienny, Daniel H. Thompson},
        title = {SaLinA: Sequential Learning of Agents},
        year = {2021},
        publisher = {Arxiv},salina_cl
        howpublished = {\url{https://gitHub.com/facebookresearch/salina}},
    }

News

• February 2023:
  • Adding `salina_cl, a continual learning framework using salina. See salina_cl
• November 2021:
  • Complete core documentation: Read the docs
• October 2021:
  • Include Decision Transformers
  • Include "A Simple Way to Initialize Recurrent Networks of Rectified Linear Units"
  • FIX: To avoid compatibility issues with openAI gym, the salina/agents/gym.py file as been renamed as salina/agents/gyma.py
  • Updated documentation
  • Transformer-based Agents
  • Behavioral Cloning over complete episodes with and without transformers.
  • PPO on Brax with Transformers
  • Release of the library
  • Various bug fixes
  • Add train and eval mode in the RemoteAgent and NRemoteAgent
  • Cleaning of the BRAX PPO Implementation to make it similar to other implementations
  • Cleaning of the Behavioral Cloning implementation

Quick Start

• Just clone the repo
• pip install -e .

Documentation

• Read the docs
• Principles of the library
• Examples and Algorithms
• Tutorial through multiple A2C implementations
• Reinforcement Learning Benchmark
• Video Tutorials:
  • Tutorial 1: Agent and Workspace
  • Tutorial 2: Multi-CPUs
  • Tutorial 3: Implementing A2C
  • Tutorial 4: A2C on multi-CPUs
• Arxiv Paper

A note on transformers

We include both classical pytorch transformers, and xformers-based implementations. On the Behavioral Cloning examples xformers-based models perform faster than classical pytorch implementations since they benefit from the use of sparse attention. Here is a small table describing the obtained results.

n transitions (in the past)	5	10	50	100	All previous transitions (episode size is 1000 transitions)
xformers	1200K / 2.3Gb	1000K / 2.3 Gb	890K / 2.5 Gb	810K / 2.1 Gb	390K / 3.8 Gb
pytorch	460K / 4.5 Gb	460K / 4.5 Gb	460K / 4.5 Gb	460K / 4.5 Gb	460K / 4.5 Gb

The table contains the number of transitions processed per second (during learning) and the memory used (using GPU)

For development, set up pre-commit hooks:

• Run pip install pre-commit
  • or conda install -c conda-forge pre-commit
  • or brew install pre-commit
• In the top directory of the repo, run pre-commit install to set up the git hook scripts
• Now pre-commit will run automatically on git commit!
• Currently isort, black and blacken-docs are used, in that order

Organization of the repo

• salina is the core library
  • salina.agents is the catalog of agents (the same as torch.nn but for agents)
• salina_examples provide many examples (in different domains)

Dependencies

salina utilizes pytorch, hydra for configuring experiments, and gym for reinforcement learning algorithms.

Note on the logger

We provide a simple Logger that logs in both tensorboard format, but also as pickle files that can be re-read to make tables and figures. See logger. This logger can be easily replaced by any other logger.

Description

Sequential Decision Making is much more than Reinforcement Learning

• Sequential Decision Making is about interactions:
• Interaction with data (e.g attention-models, decision tree, cascade models, active sensing, active learning, recommendation, etc….)
• Interaction with an environment (e.g games, control)
• Interaction with humans (e.g recommender systems, dialog systems, health systems, …)
• Interaction with a model of the world (e.g simulation)
• Interaction between multiple entities (e.g multi-agent RL)

What salina is

• A sandbox for developping sequential models at scale.

• A small (300 hundred lines) 'core' code that defines everything you will use to implement agents involved in sequential decision learning systems.

  • It is easy to understand and use since it keeps the main principles of pytorch, just extending nn.Module to Agent in order to handle the temporal dimension

• A set of agents that can be combined (like pytorch modules) to obtain complex behaviors

• A set of references implementations and examples in different domains Reinforcement Learning, Imitation Learning, Computer Vision, with more to come...

What salina is not

• Yet another reinforcement learning framework: salina is focused on sequential decision making in general. It can be used for RL (which is our main current use-case), but also for supervised learning, attention models, multi-agent learning, planning, control, cascade models, recommender systems, among other use cases.
• A library: salina is just a small layer on top of pytorch that encourages good practices for implementing sequential models. Accordingly, it is very simple to understand and use, while very powerful.

Papers using SaLinA:

• Learning a subspace of policies for online adaptation in Reinforcement Learning. Jean-Baptiste Gaya, Laure Soulier, Ludovic Denoyer - Arxiv
• Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching. Pierre-Alexandre Kamienny, Jean Tarbouriech, Alessandro Lazaric, Ludovic Denoyer - Arxiv

License

salina is released under the MIT license. See LICENSE for additional details about it. See also our Terms of Use and Privacy Policy.