Reinforcement Learning Intermediate

Reinforcement learning (RL) trains intelligent agents that learn optimal strategies through interaction with the environment. For networking, RL agents can discover routing policies, load balancing strategies, and resource allocation schemes that outperform hand-crafted rules.

RL Concepts for Networking

Q-Learning for Routing

Q-learning maintains a table of state-action values. The agent learns which actions yield the best long-term rewards in each state.

import numpy as np

# Simple Q-learning for path selection (3 paths between A and B)
n_states = 10   # discretized network load levels
n_actions = 3   # 3 possible paths
Q = np.zeros((n_states, n_actions))
alpha = 0.1      # learning rate
gamma = 0.95     # discount factor
epsilon = 0.1    # exploration rate

def choose_path(state):
    if np.random.random() < epsilon:
        return np.random.randint(n_actions)  # explore
    return np.argmax(Q[state])  # exploit

def update_q(state, action, reward, next_state):
    best_next = np.max(Q[next_state])
    Q[state, action] += alpha * (reward + gamma * best_next - Q[state, action])

Deep RL for Complex Networks

For large networks with continuous state spaces, Deep Q-Networks (DQN) and policy gradient methods replace the Q-table with neural networks:

• DQN - Neural network approximates Q-values. Good for discrete action spaces (e.g., path selection)
• PPO/A3C - Policy gradient methods for continuous action spaces (e.g., bandwidth allocation)
• Multi-Agent RL - Multiple agents controlling different network segments cooperatively

Practical Applications

• Traffic Engineering - RL optimizes ECMP weights or SR-TE policies to minimize congestion
• Load Balancing - Agent learns to distribute requests across servers based on response times
• Resource Allocation - Dynamic allocation of VLAN, QoS, and bandwidth resources
• Network Slicing (5G) - Allocate resources to virtual network slices dynamically