Every major language model released in the last few years, from GPT to Claude to Gemini, is built on the same core idea: the transformer. Introduced in the 2017 paper “Attention Is All You Need” by Vaswani et al., the architecture replaced the recurrent neural networks (RNNs) and LSTMs that had dominated NLP for a decade. Understanding why it won is the fastest way to understand why modern AI looks the way it does, and why so much of the engineering effort in frontier AI labs today goes into making this same core mechanism faster and cheaper to run rather than replacing it.
The Problem Transformers Solved
RNNs process text one token at a time, carrying a hidden state forward from word to word. That sequential dependency is the problem: you can’t compute token 50 until you’ve computed tokens 1 through 49, which makes RNNs slow to train and bad at connecting information across long distances. By the time an RNN reaches the end of a long paragraph, information from the first sentence has often faded from its hidden state, a limitation researchers called the “vanishing gradient” problem, since the gradient signal used to update early-layer weights shrinks as it’s backpropagated through many sequential steps.
Transformers throw out the sequential constraint entirely. Every token in the input can attend to every other token directly, in parallel, regardless of distance. That single change is what made it practical to train models on the scale of GPT-3 and beyond; without parallelizable training, today’s LLMs would take years to train instead of weeks. The original paper’s own benchmark made the case concretely: their base transformer model reached a better English-to-German translation score than the best prior RNN-based systems while training in a fraction of the time, using a fraction of the compute.
Self-Attention
Self-attention is the mechanism that lets each token look at every other token and decide how relevant each one is. For every token, the model computes three vectors: a query (what this token is looking for), a key (what this token offers), and a value (the actual content it contributes if attended to).
The query of one token is compared against the keys of all tokens (including itself) via a dot product, producing a relevance score for each pair. Those scores are passed through a softmax so they sum to 1, then used to compute a weighted average of all the value vectors. The result is a new representation of the token that has absorbed context from everywhere in the sequence, weighted by relevance.
Concretely: in the sentence “The trophy didn’t fit in the suitcase because it was too big,” resolving what “it” refers to requires connecting “it” back to “trophy,” several words earlier. Self-attention gives every token a direct path to do exactly that, rather than relying on information surviving a long chain of sequential updates.
It helps to walk through the arithmetic in miniature. Imagine the query vector for “it” produces raw similarity scores against every other token’s key vector, say roughly 2.1 against “trophy,” 0.3 against “suitcase,” and near zero against everything else. Running those numbers through a softmax turns them into a clean probability distribution, something like 0.71 for “trophy,” 0.12 for “suitcase,” and a handful of near-zero values that round out to 1.0. The new representation the model produces for “it” is then built by taking 71% of the value vector for “trophy,” 12% of the value vector for “suitcase,” and small contributions from everything else, in effect, the model has resolved the pronoun by mostly copying forward the representation of the noun it refers to. Real trained models don’t produce numbers this clean, and this is a simplified illustration rather than an actual model’s internals, but it’s exactly the category of operation happening at every layer, for every token, simultaneously, across billions of parameters.
Multi-Head Attention
A single attention computation captures one kind of relationship. Multi-head attention runs several attention computations in parallel, each with its own learned query/key/value projections, then concatenates the results. In practice this lets different heads specialize: some heads track syntactic relationships (subject to verb), others track coreference (pronoun to noun), others pick up longer-range topical associations. GPT-style models typically use anywhere from 12 to 96+ heads per layer depending on model size, and visualizing head activations is one of the more common ways researchers probe what a trained model has actually learned.
Efficient Attention: Why Inference Speed Became Its Own Research Problem
Textbook multi-head attention, as described in the original paper, assumes every head keeps its own full set of key and value vectors. That’s fine for training, but it becomes a real memory bottleneck at inference time: serving a long conversation means caching a key/value pair for every previous token, for every head, for every layer, and that cache grows linearly with conversation length. Two separate lines of work attack this problem from different angles, and both now show up in essentially every current-generation model’s technical report.
Multi-Query Attention (MQA) shares a single set of key/value vectors across all attention heads, keeping only the query projections separate per head. This shrinks the KV cache dramatically, but early implementations showed a measurable quality hit, since every head is now forced to work from the same shared key/value representation instead of its own. Grouped-Query Attention (GQA), introduced by Google Research (Ainslie et al., 2023), splits the difference: heads are divided into a handful of groups, with each group sharing one key/value set, recovering most of MQA’s memory savings while staying much closer to full multi-head attention’s output quality. GQA is now the standard choice in most current open-weight and frontier LLMs.
Separately, FlashAttention (Dao et al., 2022) doesn’t change what attention computes, it changes how it’s computed on the underlying GPU hardware. Naive attention implementations write the full attention score matrix out to GPU memory and read it back for the softmax and weighted-sum steps; for long sequences that matrix is enormous, and the memory traffic, not the actual arithmetic, becomes the bottleneck. FlashAttention restructures the computation to keep intermediate results in fast on-chip memory and never materializes the full matrix, producing mathematically identical output while running substantially faster and using less memory. It’s now the default attention implementation across essentially every major training and inference framework, which is a large part of why context windows have grown from a few thousand tokens to hundreds of thousands without a proportional blowup in serving cost.
Positional Encoding
Because self-attention treats the input as a set rather than a sequence, it has no inherent notion of word order, “dog bites man” and “man bites dog” would look identical to raw self-attention. Positional encoding fixes this by injecting information about each token’s position into its embedding before the attention layers see it.
The original transformer paper used fixed sinusoidal functions of different frequencies for this, which have the convenient property of letting the model generalize to sequence lengths it wasn’t trained on, at least in theory. Most current large language models instead use relative positional schemes like RoPE (rotary position embedding), introduced in the RoFormer paper (Su et al., 2021), which encode the distance between tokens rather than their absolute position and tend to extrapolate better to long context windows in practice. The jump from GPT-style models handling a few thousand tokens to current frontier models handling hundreds of thousands is due in large part to better positional encoding schemes, combined with the efficient-attention techniques above, not just more compute.
| Scheme | How it encodes position | Main advantage | Typical use |
|---|---|---|---|
| Sinusoidal (original) | Fixed sine/cosine functions added to token embeddings | Simple, no extra learned parameters | Original 2017 transformer |
| Learned absolute | A trainable embedding vector per position index | Can fit patterns specific to the training data | Early BERT/GPT-2-era models |
| RoPE (rotary) | Rotates query/key vectors by an angle proportional to position | Encodes relative distance directly; strong long-context extrapolation | Most current LLMs |
| ALiBi | Adds a distance-proportional penalty directly to attention scores | No extra parameters; cheap; strong length extrapolation (Press et al., 2021) | Some long-context and efficiency-focused models |
The Rest of the Block
A full transformer layer is more than attention. Each block also includes a feed-forward network (two linear layers with a nonlinearity between them, applied identically to every token position), residual connections around both the attention and feed-forward sublayers, and layer normalization. The residual connections matter more than they sound: without a direct path for gradients to flow around each sublayer, networks this deep (modern LLMs stack dozens to over a hundred of these blocks) would be nearly impossible to train, since the gradient signal would have to survive backpropagating through every single sublayer in sequence.
Why This Architecture Spread Beyond Text
The transformer’s core trick, treating an input as a set of tokens that attend to each other, generalizes to anything you can chop into discrete pieces. Vision Transformers (ViT), introduced by Dosovitskiy et al. in “An Image Is Worth 16x16 Words”, split images into fixed-size patches and treat each patch as a token, and now compete with or beat the convolutional networks covered in our computer vision fundamentals explainer on many large-scale vision benchmarks. Protein-folding models like DeepMind’s AlphaFold use attention-based components over amino acid sequences and pairwise residue representations to predict 3D protein structure with near-experimental accuracy, a result published in Nature in 2021 and widely credited with reshaping structural biology research. Audio models tokenize waveforms the same way. The architecture that started as an NLP paper is now the default backbone across most of deep learning, which is part of why “transformer” and “attention” show up constantly in AI coverage even outside of language models.
What This Means in Practice
If you’re evaluating or building with LLMs, a few practical consequences fall directly out of the architecture:
- Context length costs scale quadratically with naive self-attention (every token attends to every other token), which is why long-context models rely on optimizations like FlashAttention, GQA, sparse attention, or sliding windows rather than brute-forcing the full quadratic computation.
- Attention is why LLMs can do in-context learning, picking up a pattern from a few examples in the prompt without any weight updates, because attention lets the model directly reference those examples when generating the next token.
- Positional encoding choice is why some models handle long documents better than others even at similar parameter counts; it’s a common differentiator between model families that’s easy to overlook when comparing benchmark scores alone.
- KV-cache size, driven by attention variant (full multi-head vs. GQA vs. MQA), directly determines how many concurrent users a given amount of GPU memory can serve, which is why attention efficiency work gets as much engineering attention at frontier labs as raw model capability.
The mechanism is a few pages of matrix multiplication, but it’s the single architectural idea underlying essentially every major AI model release you’ll read about on this site. For a look at how these architectural choices trade off against pure compute scale, see our explainer on neural network optimization techniques.
