Key Papers

A guided walkthrough of the most influential AI papers, starting with "Attention Is All You Need" - the paper that launched the modern AI revolution.

Attention Is All You Need (2017)

Vaswani et al. - arXiv:1706.03762

This paper introduced the Transformer architecture, which replaced recurrent neural networks (RNNs) and became the foundation of GPT, BERT, and virtually all modern language models.

Key Contributions

• Self-attention mechanism: Allows every token to attend to every other token in the sequence, regardless of distance.
• Multi-head attention: Runs multiple attention operations in parallel, letting the model focus on different types of relationships.
• Positional encoding: Since transformers have no inherent notion of order, sinusoidal position encodings inject sequence position information.
• Parallelization: Unlike RNNs which process tokens sequentially, transformers process all tokens simultaneously, enabling massive speedups on GPUs.

The Self-Attention Formula

Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) * V

Where:
  Q = Query matrix  (what am I looking for?)
  K = Key matrix    (what do I contain?)
  V = Value matrix  (what information do I carry?)
  d_k = dimension of keys (scaling factor)

Essential Reading List

How to Approach a Landmark Paper

1. Read a Blog Post First

   For foundational papers, excellent blog posts and video explanations exist. Use them to build intuition before reading the paper itself.

2. Study the Architecture Diagram

   Spend time on Figure 1. Trace the data flow from input to output. Label each component.

3. Focus on One Key Equation

   Identify the paper's core equation (e.g., the attention formula). Understand it deeply. The rest of the math typically follows from this.

4. Read the Ablation Study

   Ablation studies remove or modify components one at a time, showing which parts actually matter. This reveals the true contributions.