IMRLEnv (Intersection Management Reinforcement Learning Environment)

This project is Reinforcement Learning Environment to make optimized agent controls intersection with traffic lights

Table

• IMRLEnv (Intersection Management Reinforcement Learning Environment)
  • Table
  • How to execute
  • How this environment works
    • Assets used
    • Environment
    • Scripts
    • modified iTS scripts
    • Python Agents
    • Total Flow of Environment
  • Platform Setting
  • Changes

How to execute

Clone this repository. Then, inside ./agents, execute IMRLEnv_main.py like this

python IMRLEnv_main.py --platform PLATFORM_YOU_WANT

at PLATFORM_YOU_WANT, put windows-desktop to use windows desktop environment, put windows-server to use windows headless environment, put linux-desktop to use linux desktop environment.

Linux headless environment is default, so you don't need to enter anything at --platform and execute like this:

python3 IMRLEnv_main.py

Arguments may added further.

How this environment works

Assets used

• Unity ml-agent: Implement RL agents and RL environment
• iTS - Intelligent Traffic System(Source): Implement car traffic
• Free Low Poly Simple Urban City 3D Asset Pack: Used in appearance of environment

Environment

• Environment Overview

  • Environment is four-way intersection.
  • Each road have 6 lanes. Each lane has specific directions(left turn, straight, right turn).
  • Each lane has traffic light with red, yellow, green light.

• Observation : vector observation

  • Traffic Light State : one-hot encoding : 3*12
  • Vehicles average waiting time by lanes descripted with 0~3 : 12
  • Total observation size : 48

• Action : Discrete

  • Pre-defined action : if action has conflicting another traffic light state, TR Manager will change that state
  • If there are no conflicting Traffic Light State(right turn), action can turn on&off green light
  • Total action size : 12

• Reward

  • At every step, "diffence of average waiting time" will given as reward
  • "diffence of average waiting time" = sum of normalized [(allowed wait time) - (average waiting time by lane)]^2

• End of Episode

  • When certain amount of vehicles pass through Intersection
  • When "total average waiting time" exceed "allowed waiting time"

• At training mode, time scale will be 12.

Scripts

• IMAgent.cs: Main agent. Handle actions from python agent, send rewards to python agent.
• IMSetting.cs: Do environment reset, display environment infos at scene.
• ModifiedSpawnerManager.cs: handles TSTrafficSpawner.cs
• TRManager.cs: handles TSTrafficLight.cs
• IMCarTracker.cs: gather informations from DetectorColider.cs and return values need to calculate reward.
• DetectorColider.cs: colider scripts

modified iTS scripts

TSTrafficSpawner.cs and TSTrafficLight.cs inside iTS asset need to be modified to make environment works.

TSTrafficSpawner.cs is modified to reset environment when episode ends. TSTrafficLight.cs is modified to turn on/off lights by commands it get.

Python Agents

• Random Agent
• DQN based Agent(code is done, specific parameters needed to update)

Total Flow of Environment

First image shows flow when python agent called env.reset(). After env.reset() called, IMAgent calles OnEpisodeBegin() and IMSetting reset environments.

Second image shows flow when env.set_action() called from python agent. When Action send to IMAgent, IMAgent handle traffic lights via TRManager.

Last image shows how reward is given to python agent. Every step, IMCarTracker returns values(in this case, "average waiting time" and "average traveling time") needed at calculating reward. Then, IMAgent calculates reward and send it back to python agent. IMAgent also send information about "end of episode".

Platform Setting

There are four version of the env

• Linux Desktop : not tested
• Linux Headless : not tested
• Windows Desktop : tested with Random Agent, working
• Windows Headless : tested with Random Agent, working

Also, this environment was tested in these conditions

• both Linux and Windows 11

  • Unity ML-Agent (unity) : 2.0.0
  • mlagents, mlagents-env : 0.26.0

• At Windows 11

  • python 3.7.16 working on anaconda
  • torch : 1.8.0+cu111
  • CUDA : 11.8

• At Linux(Ubuntu 20.04)

  • python 3.8.17 working on anaconda
  • protobuf : 3.20.0
  • CUDA : 11.4

Changes

• 2023.10.24 : added infoFile.txt to change parameters inside IMRLEnv
• 2023.09.21 : updated reward function / changed DQN's Q function with more complex network
• 2023.09.20 : made IMRLEnv_data.py to handle parameters / changed environment parameters for better scoring
• 2023.09.19 : changed time scale will be 12, DQN code is finished.
• 2023.09.14 : changed reward with "diffence of average waiting time", made server version app displays information of env in console at the begining of episode.
• 2023.09.13 : made DQN agnet(not confirmed) and changed folder names to run IMRLEnv_main.py easily
• 2023.08.21 : solved environment reset problem, uploaded several different platform version of IMRLEnv, add argument option to select platform.
• 2023.08.17 : first commit