Q-Learning

Q-learning is a value-based, model-free Reinforcement learning algorithm where the agent learns the optimal policy by updating Q-values based on the rewards received. It is particularly useful in discrete environments like grids.

Uses a Q-Table which is populated by Q-values which are the maximum expected future reward for the given state and action. We improve the Q-Table in an iterative approach

Resources:

• Q-Learning Explained - Reinforcement Learning Tutorial

Q-learning update rule:

The left hand side gets updated (Bellman Equations)

$$Q_{new}(s_t,a_t)\leftarrow Q(s_t,a_t)+\alpha \left[r_{t+1}+\gamma \underset{a^{\prime }}{\max }Q(s_{t+1},a^{\prime })-Q(s_t,a_t)\right]$$

Explanation:

• $Q(s_t,a_t)$: The Q-value of the current state $s_t$ and action $a_t$.
• $\alpha$: The learning rate, determining how much new information overrides old information.
• $r_{t+1}$: The reward received after taking action $a_t$ from state $s_t$.
• $\gamma$: The discount factor, balancing immediate and future rewards.
• ${\max }_{a^{\prime }}Q(s_{t+1},a^{\prime })$: The maximum Q-value for the next state $s_{t+1}$ across all possible actions $a^{\prime }$.

Notes:

• Q-learning is well-suited for environments where the state and action spaces are discrete and manageable in size.
• The algorithm is designed to converge to the optimal policy, even in non-deterministic environments, as long as each state-action pair is sufficiently explored.
• Exploration vs. Exploitation