RL Applications

From mastering board games to aligning language models, reinforcement learning powers some of the most impressive AI systems in the world.

Game Playing

Games have been the primary testbed for RL research because they provide well-defined rules, clear reward signals, and measurable performance:

• Atari (DQN): A single DQN agent learned to play 49 Atari games from raw pixels, surpassing human-level performance on 29 of them.
• Go (AlphaGo/AlphaZero): Combined Monte Carlo tree search with deep RL. AlphaZero learned Go, chess, and shogi entirely through self-play, surpassing all previous AI systems.
• StarCraft II (AlphaStar): DeepMind's agent reached Grandmaster level, handling partial observability, long-time horizons, and massive action spaces.
• Dota 2 (OpenAI Five): Five cooperating RL agents defeated the world champion team, demonstrating multi-agent coordination.

from stable_baselines3 import PPO
from stable_baselines3.common.atari_wrappers import AtariWrapper
from stable_baselines3.common.vec_env import DummyVecEnv, VecFrameStack
import gymnasium as gym

def make_env():
    env = gym.make("ALE/Breakout-v5")
    env = AtariWrapper(env)
    return env

# Vectorized environment with frame stacking
env = DummyVecEnv([make_env])
env = VecFrameStack(env, n_stack=4)

# Train PPO agent on Atari Breakout
model = PPO("CnnPolicy", env, verbose=1, n_steps=128, batch_size=256)
model.learn(total_timesteps=1_000_000)
model.save("ppo_breakout")

Robotics

RL enables robots to learn complex behaviors through trial and error:

• Manipulation: Robotic arms learning to grasp objects, open doors, and assemble parts. OpenAI's Rubik's cube solver learned entirely in simulation.
• Locomotion: Quadruped and humanoid robots learning to walk, run, and navigate rough terrain using SAC and PPO.
• Sim-to-Real Transfer: Train in simulation (fast, safe, cheap) and transfer the policy to a real robot. Domain randomization helps bridge the reality gap.

RLHF: Aligning Language Models

Reinforcement Learning from Human Feedback (RLHF) is the technique that transformed GPT-3 into ChatGPT:

1. Supervised Fine-Tuning

   Fine-tune a pretrained LLM on human-written demonstration data.

2. Train Reward Model

   Humans rank different model outputs. A reward model learns to predict human preferences.

3. RL Optimization

   Use PPO to fine-tune the LLM to maximize the reward model's score while staying close to the original model (KL penalty).

Other Applications

OpenAI Gym / Gymnasium

Gymnasium (formerly OpenAI Gym) is the standard API for RL environments. It provides a consistent interface for hundreds of environments:

import gymnasium as gym

# Classic control (simple, fast)
env = gym.make("CartPole-v1")       # Balance a pole
env = gym.make("MountainCar-v0")    # Drive up a hill
env = gym.make("LunarLander-v3")    # Land a spacecraft

# Box2D (2D physics)
env = gym.make("BipedalWalker-v3")  # 2D walking robot

# Atari (pixel-based games)
env = gym.make("ALE/Pong-v5")       # Classic Pong

# MuJoCo (3D physics, continuous control)
env = gym.make("Humanoid-v4")       # 3D humanoid locomotion

# Standard interface
obs, info = env.reset()
action = env.action_space.sample()
obs, reward, terminated, truncated, info = env.step(action)