Understanding Transformers

In 2017, eight Google researchers published a paper with a deceptively modest title: "Attention Is All You Need." The architecture they introduced — the Transformer — didn't just improve the state of the art. It obliterated it, and then went on to reshape the entire field of AI. This is how it works.

The Problem Transformers Solved#

Before Transformers, sequence modelling relied on recurrent neural networks (RNNs). RNNs process tokens one at a time, left to right. This has two fatal flaws:

1. They can't parallelise — you can't process token 5 until you've finished tokens 1–4
2. Long-range dependencies vanish — by the time the model reaches token 100, it has largely forgotten token 1

The Transformer solves both problems with a single mechanism: attention.

The Attention Mechanism#

The core idea: instead of reading tokens sequentially, let every token look at every other token simultaneously and decide which ones are relevant.

For a token $q$ (the "query"), attention is computed as:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$

Where:

• $Q$ = Query matrix (what am I looking for?)
• $K$ = Key matrix (what does each token offer?)
• $V$ = Value matrix (what information do I retrieve?)
• $d_k$ = dimension of the keys (a scaling factor to prevent vanishing gradients)

The $\sqrt{d_k}$ in the denominator is crucial. Without it, the dot products grow large in high dimensions, pushing the softmax into regions with near-zero gradients.

Multi-Head Attention#

A single attention head learns one type of relationship. Multi-head attention runs $h$ attention heads in parallel, each learning different relationships:

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model: int, num_heads: int):
        super().__init__()
        self.num_heads = num_heads
        self.d_k = d_model // num_heads
        
        self.W_q = nn.Linear(d_model, d_model)
        self.W_k = nn.Linear(d_model, d_model)
        self.W_v = nn.Linear(d_model, d_model)
        self.W_o = nn.Linear(d_model, d_model)
    
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        B, T, C = x.shape
        
        # Project to Q, K, V and split into heads
        Q = self.W_q(x).view(B, T, self.num_heads, self.d_k).transpose(1, 2)
        K = self.W_k(x).view(B, T, self.num_heads, self.d_k).transpose(1, 2)
        V = self.W_v(x).view(B, T, self.num_heads, self.d_k).transpose(1, 2)
        
        # Scaled dot-product attention
        scores = (Q @ K.transpose(-2, -1)) / (self.d_k ** 0.5)
        attn = scores.softmax(dim=-1)
        out = attn @ V
        
        # Concatenate heads and project
        out = out.transpose(1, 2).contiguous().view(B, T, C)
        return self.W_o(out)

The Full Architecture#

The Transformer is an encoder-decoder architecture. Here's a high-level picture:

Each encoder block contains:

1. Multi-head self-attention
2. Feed-forward network
3. Layer norm + residual connections after each

Each decoder block contains:

1. Masked multi-head self-attention (can't peek at future tokens)
2. Cross-attention over encoder output
3. Feed-forward network

Modern LLMs like GPT-4 usedecoder-only Transformers — no encoder. The model simply predicts the next token, autoregressively.

Positional Encoding#

Since attention has no notion of order, Transformers need to inject position information explicitly. The original paper used fixed sinusoidal encodings:

$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)$ $PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$

Modern models use Rotary Position Embeddings (RoPE) instead, which handle much longer contexts more gracefully.

Why It Works So Well#

The Transformer's success comes from a few key properties:

Property	Benefit
Fully parallel training	Utilises GPUs to their maximum
$O(1)$ path length	Any two tokens interact directly in one layer
Scalable	More parameters → consistently better results
Transfer learning	Pretrain once on massive text, fine-tune for any task

The scaling laws paper (Kaplan et al., 2020) showed that Transformer performance follows predictable power laws with respect to model size, data, and compute. This gave labs confidence to invest billions in larger models.

Summary#

The Transformer replaced sequential processing with global attention, enabling parallel training and long-range understanding. Its elegance lies in learning which information matters, not just where it is. Every major language model today — GPT, Claude, Gemini, LLaMA — is a Transformer at its core.