Q-Learning and Double Q_learning
Budget: $8 – $15 USD
Consider the following gridworld with four actions (up, down, right, and left). If the action takes the agent off the gird, the agent stays in the same state. In each non-terminating step, the agent receives a random reward of -12 or +10 with equal probability. The reward for reaching the goal state is +5 and the episode ends when the agent reaches the goal.
Use ? –greedy policy with ?(?)= 1/√?(?) where ?(?) is the number of times state has been visited, assuring infinite exploration in the limit which is a theoretical requirement for the convergence of both Q-learning and Double Q-learning.
a. Implement Q-learning and Double Q-learning and apply them to this problem for 10,000 experiments using the learning rate ? = 1 / ?(?,?)
b. Plot the average reward per step vs. number of time steps averaged over 10,000 experiments. The length of an episode following the optimal policy is five actions, so the optimal average reward per step is +0.2. Plot this true value in your figure and see how close your algorithm gets to the true value.
c. Plot the maximal action value in the starting state ? (i.e. ?????(?,?) averaged over 10,000 experiments. The optimal value of maximally valued action in the starting state is 5?4−∑3?=0≈0.36. Plot this true value in your figure and see how close your algorithm gets to the true value.
d. Repeat the experiments with ? = 1/ ?(?,?)^0.8 and redo steps b and c.
Use ? –greedy policy with ?(?)= 1/√?(?) where ?(?) is the number of times state has been visited, assuring infinite exploration in the limit which is a theoretical requirement for the convergence of both Q-learning and Double Q-learning.
a. Implement Q-learning and Double Q-learning and apply them to this problem for 10,000 experiments using the learning rate ? = 1 / ?(?,?)
b. Plot the average reward per step vs. number of time steps averaged over 10,000 experiments. The length of an episode following the optimal policy is five actions, so the optimal average reward per step is +0.2. Plot this true value in your figure and see how close your algorithm gets to the true value.
c. Plot the maximal action value in the starting state ? (i.e. ?????(?,?) averaged over 10,000 experiments. The optimal value of maximally valued action in the starting state is 5?4−∑3?=0≈0.36. Plot this true value in your figure and see how close your algorithm gets to the true value.
d. Repeat the experiments with ? = 1/ ?(?,?)^0.8 and redo steps b and c.