Introduction to Reinforcement Learning

Reinforcement Learning (RL) is a paradigm where agents learn optimal behavior by interacting with an environment and receiving rewards or penalties.

What is Reinforcement Learning?

Reinforcement Learning is a type of machine learning where an agent learns to make decisions by performing actions in an environment and observing the resulting rewards. Unlike supervised learning (which requires labeled examples) or unsupervised learning (which finds patterns in data), RL learns from the consequences of its own actions.

Think of training a dog: you don't show it thousands of labeled examples of "sit." Instead, the dog tries different actions, and when it sits on command, you give it a treat (reward). Over time, it learns which actions lead to treats.

The RL Framework

Every RL problem has four key components:

• Agent: The learner and decision-maker. It observes the environment, takes actions, and receives rewards.
• Environment: Everything the agent interacts with. It receives actions from the agent and returns new states and rewards.
• State (s): A representation of the current situation. In a chess game, the state is the board position.
• Action (a): A choice the agent makes. In chess, an action is a move.
• Reward (r): A scalar feedback signal indicating how good the action was. Positive rewards encourage behavior; negative rewards discourage it.
• Policy (π): The agent's strategy - a mapping from states to actions. The goal is to find the optimal policy.

import gymnasium as gym

# Create an environment
env = gym.make("CartPole-v1")

# Reset to initial state
state, info = env.reset()

for step in range(1000):
    # Agent selects an action (random for now)
    action = env.action_space.sample()

    # Environment returns: next_state, reward, terminated, truncated, info
    next_state, reward, terminated, truncated, info = env.step(action)

    # Agent learns from this experience
    # learn(state, action, reward, next_state)

    state = next_state

    if terminated or truncated:
        state, info = env.reset()

env.close()

RL vs Supervised vs Unsupervised Learning

Key Challenges in RL

• Exploration vs Exploitation: Should the agent try new actions (explore) or stick with what it knows works (exploit)? Too much exploration wastes time; too much exploitation misses better strategies.
• Delayed Rewards: In chess, the reward (win/loss) comes only at the end. The agent must figure out which earlier moves contributed to the outcome - this is the credit assignment problem.
• Sample Efficiency: RL often needs millions of interactions to learn. Making agents learn faster from fewer experiences is an active research area.
• Stability: RL training can be unstable. Small changes in hyperparameters can cause completely different behaviors.

Landmark RL Achievements

1. TD-Gammon (1992)

   Gerald Tesauro's backgammon agent learned to play at expert level using temporal difference learning and a neural network.

2. Atari DQN (2013)

   DeepMind's DQN learned to play Atari games from raw pixels, achieving superhuman performance on many games with a single algorithm.

3. AlphaGo (2016)

   DeepMind's AlphaGo defeated world champion Go player Lee Sedol, combining RL with Monte Carlo tree search and deep neural networks.

4. OpenAI Five (2019)

   A team of five RL agents defeated the world champion Dota 2 team, demonstrating multi-agent coordination in complex environments.

5. ChatGPT & RLHF (2022)

   Reinforcement Learning from Human Feedback (RLHF) became the key technique for aligning large language models with human preferences.