RNNs & LSTMs

Explore Recurrent Neural Networks for sequential data processing, understand the vanishing gradient problem, and learn how LSTMs and GRUs solve it.

Recurrent Neural Networks

Standard neural networks process each input independently. But many tasks involve sequential data where order matters: text (word by word), speech (frame by frame), time series (step by step). Recurrent Neural Networks (RNNs) address this by maintaining a hidden state that acts as memory, carrying information from previous time steps.

# At each time step t:
h_t = tanh(W_hh * h_(t-1) + W_xh * x_t + b_h)  # Update hidden state
y_t = W_hy * h_t + b_y                            # Compute output

# h_t: hidden state at time t (the "memory")
# x_t: input at time t
# y_t: output at time t
# W_hh, W_xh, W_hy: shared weight matrices

The key idea is weight sharing: the same weights are used at every time step. This allows RNNs to handle sequences of any length and generalize patterns across different positions in the sequence.

The Vanishing Gradient Problem

In theory, RNNs can learn long-range dependencies. In practice, they struggle because of the vanishing gradient problem. During backpropagation through time (BPTT), gradients are multiplied at each time step. If these multiplied values are less than 1, the gradient shrinks exponentially, becoming nearly zero for early time steps.

Long Short-Term Memory (LSTM)

LSTMs, introduced by Hochreiter and Schmidhuber in 1997, solve the vanishing gradient problem using a sophisticated gating mechanism. An LSTM cell has three gates that control information flow:

1. Forget Gate

   Decides what information to discard from the cell state. Uses a sigmoid function that outputs values between 0 (forget completely) and 1 (keep entirely). Example: when processing a new subject in a sentence, forget the old subject's gender.

2. Input Gate

   Decides what new information to store in the cell state. A sigmoid layer decides which values to update, and a tanh layer creates candidate values. Example: store the new subject's gender to use later for pronoun agreement.

3. Output Gate

   Decides what parts of the cell state to output as the hidden state. A sigmoid layer controls which parts of the cell state are exposed. Example: when predicting a verb, output the relevant subject information.

The cell state acts like a conveyor belt running through the entire sequence. Information can flow along it unchanged, with gates selectively adding or removing information. This direct path for gradients solves the vanishing gradient problem.

Gated Recurrent Units (GRU)

GRUs, introduced by Cho et al. in 2014, are a simplified version of LSTMs that merge the forget and input gates into a single update gate and combine the cell state and hidden state. They have fewer parameters and are faster to train:

Bidirectional RNNs

Standard RNNs only process sequences in one direction (left to right). Bidirectional RNNs process the sequence in both directions simultaneously, capturing context from both past and future. The outputs from both directions are concatenated at each time step.

This is especially useful for tasks where the full context matters, such as named entity recognition ("Apple" could be a fruit or a company - the surrounding words help disambiguate).

Applications

• Text generation: Character-by-character or word-by-word text generation.
• Machine translation: Encoder-decoder RNNs for sequence-to-sequence tasks (largely replaced by Transformers).
• Speech recognition: Converting audio waveforms to text using bidirectional LSTMs.
• Time series forecasting: Predicting stock prices, weather, or sensor readings.
• Sentiment analysis: Understanding the sentiment of a sentence or document.
• Music generation: Generating melodies and harmonies.

Code Example: LSTM for Text Classification

import torch
import torch.nn as nn

class LSTMClassifier(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim,
                 num_classes, num_layers=2):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTM(
            input_size=embed_dim,
            hidden_size=hidden_dim,
            num_layers=num_layers,
            batch_first=True,
            bidirectional=True,   # Bidirectional
            dropout=0.3
        )
        # *2 because bidirectional
        self.fc = nn.Linear(hidden_dim * 2, num_classes)
        self.dropout = nn.Dropout(0.5)

    def forward(self, x):
        # x shape: (batch_size, seq_length)
        embedded = self.dropout(self.embedding(x))
        # embedded: (batch_size, seq_length, embed_dim)

        lstm_out, (hidden, cell) = self.lstm(embedded)
        # Use last hidden states from both directions
        hidden = torch.cat((hidden[-2], hidden[-1]), dim=1)
        output = self.fc(self.dropout(hidden))
        return output

# Initialize the model
model = LSTMClassifier(
    vocab_size=10000,
    embed_dim=128,
    hidden_dim=256,
    num_classes=2   # Binary: positive/negative
)

# Example forward pass
sample_input = torch.randint(0, 10000, (32, 50))  # batch=32, seq=50
output = model(sample_input)
print(output.shape)  # torch.Size([32, 2])