Reinforcement Learning: Race Track
Budget: $10 – $30 USD
We have implemented a custom environment called "Racetrack". you are required to implement an agent which learns to reach a goal state in the racetrack task using On-Policy First-Visit MC Control, Sarsa algorithm, Q-Learning algorithm, Modified Q-Learning Agent, and a simple Comparison & Discussion.
the implementation will be simple and all instructions are given in a jupyter notebook.
Environment Description
Consider driving a race car around a turn on a racetrack. In order to complete the race as quickly as possible, you would want to drive as fast as you can but, to avoid running off the track, you must slow down while turning.
In our simplified racetrack environment, the agent is at one of a discrete set of grid positions. The agent also has a discrete speed in two directions, ? and ? . So the state is represented as follows:
(position?,position?,velocity?,velocity?)
The agent collects a reward of -1 at each time step, an additional -10 for leaving the track (i.e., ending up on a black grid square in the figure below), and an additional +10 for reaching the finish line (any of the red grid squares). The agent starts each episode on a randomly selected grid-square on the starting line (green grid squares) with a speed of zero in both directions. At each time step, the agent can change its speed in both directions. Each speed can be changed by +1, -1 or 0, giving a total of nine actions. For example, the agent may increase its speed in the ? direction by -1 and its speed in the ? direction by +1. The agent's speed cannot be greater than +10 or less than -10 in either direction.
The agent's next state is determined by its current grid square, its current speed in two directions, and the changes it makes to its speed in the two directions. This environment is stochastic. When the agent tries to change its speed, no change occurs (in either direction) with probability 0.2. In other words, 20% of the time, the agent's action is ignored and the car's speed remains the same in both directions.
If the agent leaves the track, it is returned to a random start grid-square and has its speed set to zero in both directions; the episode continues. An episode ends only when the agent transitions to a goal grid-square.
Environment Implementation
We have implemented the above environment in the racetrack_env.py file, for this project. Please use this implementation instead of writing your own, and please do not modify the environment.
We provide a RacetrackEnv class for your agents to interact with. The class has the following methods:
reset() - this method initialises the environment, chooses a random starting state, and returns it. This method should be called before the start of every episode.
step(action) - this method takes an integer action (more on this later), and executes one time-step in the environment. It returns a tuple containing the next state, the reward collected, and whether the next state is a terminal state.
render(sleep_time) - this method renders a matplotlib graph representing the environment. It takes an optional float parameter giving the number of seconds to display each time-step. This method is useful for testing and debugging, but should not be used during training since it is very slow. Do not use this method in your final submission.
get_actions() - a simple method that returns the available actions in the current state. Always returns a list containing integers in the range [0-8] (more on this later).
In our code, states are represented as Python tuples - specifically a tuple of four integers. For example, if the agent is in a grid square with coordinates ( ?=2 , ?=3 ), and is moving zero cells vertically and one cell horizontally per time-step, the state is represented as (2, 3, 0, 1). Tuples of this kind will be returned by the reset() and step(action) methods. It is worth noting that tuples can be used to index certain Python data-structures, such as dictionaries.
the implementation will be simple and all instructions are given in a jupyter notebook.
Environment Description
Consider driving a race car around a turn on a racetrack. In order to complete the race as quickly as possible, you would want to drive as fast as you can but, to avoid running off the track, you must slow down while turning.
In our simplified racetrack environment, the agent is at one of a discrete set of grid positions. The agent also has a discrete speed in two directions, ? and ? . So the state is represented as follows:
(position?,position?,velocity?,velocity?)
The agent collects a reward of -1 at each time step, an additional -10 for leaving the track (i.e., ending up on a black grid square in the figure below), and an additional +10 for reaching the finish line (any of the red grid squares). The agent starts each episode on a randomly selected grid-square on the starting line (green grid squares) with a speed of zero in both directions. At each time step, the agent can change its speed in both directions. Each speed can be changed by +1, -1 or 0, giving a total of nine actions. For example, the agent may increase its speed in the ? direction by -1 and its speed in the ? direction by +1. The agent's speed cannot be greater than +10 or less than -10 in either direction.
The agent's next state is determined by its current grid square, its current speed in two directions, and the changes it makes to its speed in the two directions. This environment is stochastic. When the agent tries to change its speed, no change occurs (in either direction) with probability 0.2. In other words, 20% of the time, the agent's action is ignored and the car's speed remains the same in both directions.
If the agent leaves the track, it is returned to a random start grid-square and has its speed set to zero in both directions; the episode continues. An episode ends only when the agent transitions to a goal grid-square.
Environment Implementation
We have implemented the above environment in the racetrack_env.py file, for this project. Please use this implementation instead of writing your own, and please do not modify the environment.
We provide a RacetrackEnv class for your agents to interact with. The class has the following methods:
reset() - this method initialises the environment, chooses a random starting state, and returns it. This method should be called before the start of every episode.
step(action) - this method takes an integer action (more on this later), and executes one time-step in the environment. It returns a tuple containing the next state, the reward collected, and whether the next state is a terminal state.
render(sleep_time) - this method renders a matplotlib graph representing the environment. It takes an optional float parameter giving the number of seconds to display each time-step. This method is useful for testing and debugging, but should not be used during training since it is very slow. Do not use this method in your final submission.
get_actions() - a simple method that returns the available actions in the current state. Always returns a list containing integers in the range [0-8] (more on this later).
In our code, states are represented as Python tuples - specifically a tuple of four integers. For example, if the agent is in a grid square with coordinates ( ?=2 , ?=3 ), and is moving zero cells vertically and one cell horizontally per time-step, the state is represented as (2, 3, 0, 1). Tuples of this kind will be returned by the reset() and step(action) methods. It is worth noting that tuples can be used to index certain Python data-structures, such as dictionaries.