Training a neural network means adjusting millions or billions of parameters to minimize a loss function, and the difference between a network that trains successfully and one that stalls out at poor accuracy is very often not the architecture, it’s the optimization choices: which gradient descent variant, how the learning rate changes over time, and whether the internal activations stay in a numerically stable range throughout training.
Gradient Descent Variants
At its core, training is gradient descent: compute how the loss changes with respect to each parameter (the gradient), then nudge every parameter a small step in the direction that reduces the loss. Plain stochastic gradient descent (SGD) does exactly this, using a mini-batch of training examples at each step rather than the whole dataset, which makes it computationally tractable at scale but means each individual step is a noisy estimate of the true gradient.
SGD with momentum improves on this by accumulating a running average of past gradients and using that to inform the current update, which smooths out the noise and helps push through flat regions or small local irregularities in the loss surface that would otherwise slow plain SGD down.
Adam (Adaptive Moment Estimation) (Kingma & Ba, 2014) is the optimizer used by default in the large majority of modern deep learning training runs, including nearly all large language model training. It maintains a per-parameter adaptive learning rate, tracking both the first moment (like momentum) and the second moment (a running estimate of the squared gradient) for every parameter individually, so parameters that have historically had large, noisy gradients get smaller effective steps, and parameters with small, consistent gradients get relatively larger ones. This adaptivity is why Adam converges reliably across a much wider range of problems with much less manual learning-rate tuning than plain SGD, which is exactly why it became the default choice, even though well-tuned SGD with momentum can sometimes reach a slightly better final result once someone has done that tuning.
A widely-used refinement, AdamW (Loshchilov & Hutter, 2017), fixes a subtle bug in how the original Adam implementation combined weight decay (a regularization technique that shrinks parameters toward zero to reduce overfitting) with its adaptive learning rates. In standard Adam, weight decay gets tangled up with the per-parameter adaptive scaling in a way that makes it behave inconsistently; AdamW decouples the two, applying weight decay directly and separately from the adaptive gradient update. AdamW is now the default optimizer specified in the training recipes of most published large language models, precisely because that decoupling makes the weight-decay hyperparameter behave predictably across different model scales.
| Optimizer | Core mechanism | Main advantage | Main tradeoff |
|---|---|---|---|
| SGD | Fixed-size step in the gradient direction | Simple, well-understood, can generalize slightly better once tuned | Needs careful, often manual learning-rate tuning |
| SGD + momentum | Accumulates a running average of past gradients | Smooths noisy updates, pushes through flat regions | Still one global learning rate for all parameters |
| Adam | Per-parameter adaptive learning rate from 1st and 2nd gradient moments | Converges reliably with little tuning, handles sparse/noisy gradients | Weight decay interacts awkwardly with adaptive scaling |
| AdamW | Adam with weight decay decoupled from the adaptive update | Predictable regularization behavior at any scale | One more hyperparameter to set correctly (decay rate) |
Batch Normalization
Deep networks have a subtle internal problem: as training updates the weights of early layers, the statistical distribution of activations flowing into later layers keeps shifting, a phenomenon the original paper called internal covariate shift. Each layer effectively has to keep re-adapting to a moving target coming from the layers before it, which slows training and makes it more sensitive to initialization and learning rate choices.
Batch normalization (Ioffe & Szegedy, 2015) addresses this directly: for each mini-batch, it normalizes the activations of a given layer to have zero mean and unit variance, then applies a learned scale and shift so the network retains the ability to represent whatever distribution is actually optimal. The original paper reported reaching the same accuracy as a comparable state-of-the-art image classifier using 14 times fewer training steps. In practice this lets networks train faster, tolerate higher learning rates, and be considerably less sensitive to how weights are initialized. It’s become close to a default component in convolutional architectures, though transformer-based architectures generally use a related but distinct technique, layer normalization (Ba et al., 2016), which normalizes across features for a single example rather than across a batch, largely because it behaves better with the variable sequence lengths and small batch sizes common in NLP training.
Learning Rate Scheduling
The learning rate, how large a step to take at each update, is one of the most consequential hyperparameters in training, and a fixed learning rate for the entire training run is rarely optimal. Too high early on prevents the model from settling anywhere useful; too low throughout wastes enormous amounts of compute converging far slower than necessary.
Most modern training runs use a schedule that changes the learning rate over time rather than a fixed value. A common pattern for large model training is warmup followed by decay: start with a very small learning rate and gradually ramp it up over the first portion of training (warmup), which prevents large, destabilizing updates while the model’s weights are still close to their random initialization, then gradually decrease it for the remainder of training, commonly following a cosine curve, a technique popularized as part of SGDR, “Stochastic Gradient Descent with Warm Restarts” (Loshchilov & Hutter, 2016). The cosine schedule lets the model make large exploratory jumps early and fine, precise adjustments as it approaches convergence. Getting this schedule wrong, particularly skipping warmup on a large model, is a well-documented way for a training run to diverge outright in the first few thousand steps, which is part of why published training recipes for large models specify their learning rate schedule in as much detail as the architecture itself.
Gradient Clipping
Deep and recurrent networks are prone to a specific failure mode: occasionally, the gradient computed for a parameter update is enormous, orders of magnitude larger than typical, because the loss landscape in some region is extremely steep. Applying that update directly can throw the network’s weights into a wildly different, often much worse, region of parameter space, undoing potentially large amounts of training progress in a single step. Pascanu, Mikolov, and Bengio’s analysis of this problem (Pascanu et al., 2012) traced it to the geometry of the loss surface in deep and recurrent networks, which features narrow, sharply-curved regions where an unclipped gradient step can overshoot dramatically, and proposed the standard fix still used today: gradient norm clipping, which rescales the gradient whenever its overall magnitude exceeds a chosen threshold, capping the size of any single update regardless of how steep the local loss surface is. Skipping gradient clipping on a large model is one of the more common causes of a training run’s loss suddenly spiking to a much worse value partway through training, sometimes recoverable, sometimes not.
Mixed Precision Training
Numerical precision is itself an optimization lever. Most deep learning research historically used 32-bit floating point (FP32) numbers for weights, activations, and gradients, but Micikevicius et al.’s mixed precision training work (Micikevicius et al., 2017) showed that most of a network’s computation can run in 16-bit floating point (FP16) instead, roughly halving memory use and taking advantage of specialized hardware, like NVIDIA’s Tensor Cores, that run FP16 matrix multiplication significantly faster than FP32. The technique keeps a master copy of the weights in full FP32 precision for the actual parameter updates (to avoid small updates rounding to zero) while doing the bulk of the forward and backward pass computation in FP16, and adds a dynamic loss-scaling step to prevent small gradient values from underflowing to zero in the lower-precision format. Mixed precision training (and its modern descendants using even lower-precision formats like BF16 and FP8) is standard practice for training any model at meaningful scale today, since the memory and speed savings compound directly into how large a model, or how large a batch size, fits on a given amount of GPU hardware, a direct link between the optimization choices in this post and the hardware choices covered in our evolution of ML hardware explainer.
Why This Is More Than Academic
For anyone reading about a new model release, optimization choices are frequently buried in the technical report but explain real differences between models: a model trained with a more carefully tuned learning rate schedule and larger effective batch size (often achieved via gradient accumulation across many GPUs) can reach a better final result from the same architecture and the same amount of data, purely from better optimization. This is also where a large fraction of the “engineering,” as opposed to pure research, effort in training frontier models actually goes: getting a training run that spans thousands of GPUs for weeks to converge reliably without diverging or wasting compute is substantially an optimization and infrastructure problem, not just an architecture design problem. A poorly chosen learning rate schedule or a missing warmup phase can waste weeks of compute on a training run that silently underperforms or diverges outright, which is why these choices, unglamorous as they are compared to a new attention variant or activation function, are treated with the same rigor as architecture design at every frontier lab. For how these training-time choices interact with the underlying hardware they run on, see our evolution of ML hardware explainer, and for how the choice of framework (PyTorch, JAX, or TensorFlow) affects which of these optimizations are easy versus painful to implement, see our comparison of modern deep learning frameworks. Many of the efficiency techniques covered here are also exactly what’s driving the next generation of architectures covered in our analysis of where neural networks are headed.
