Reinforcement learning (RL) is the branch of machine learning where an agent learns by acting in an environment and receiving feedback, rather than by studying a labeled dataset. It’s the technique behind DeepMind’s AlphaGo, most robotics control systems, and, more relevantly for anyone following current AI news, the RLHF (reinforcement learning from human feedback) step that turns a raw language model into something that follows instructions and refuses harmful requests.
The Core Loop
RL formalizes a simple loop: an agent observes a state, takes an action, and receives a reward along with a new state. The agent’s goal is to learn a policy, a mapping from states to actions, that maximizes cumulative reward over time, not just the immediate reward from the next action. Most practical RL research and experimentation happens against standardized environments, and the current standard interface for this is Gymnasium, the actively maintained successor to OpenAI’s original Gym, maintained by the Farama Foundation (gymnasium.farama.org).
That “cumulative over time” part is what makes RL genuinely different from supervised learning. A chess move that looks good right now might set up a losing position ten moves later; a reward given immediately after an action doesn’t tell the full story. RL algorithms have to solve credit assignment: figuring out which of many past actions actually deserves credit for a reward received much later.
Markov Decision Processes
The mathematical framework underlying almost all of RL is the Markov Decision Process (MDP): a set of states, a set of actions available in each state, transition probabilities (what state you end up in after taking an action), and a reward function. The “Markov” part is an assumption, that the current state contains all the information needed to decide the best action, so the agent doesn’t need to remember the full history to act optimally. This framing, along with most of the foundational algorithms discussed below, is laid out in full in Sutton and Barto’s Reinforcement Learning: An Introduction, the field’s standard reference text, which its authors make freely available online (Sutton & Barto, incompleteideas.net).
This assumption is often only approximately true in practice (a robot’s camera frame doesn’t capture everything about the physical world), which is why real systems often augment the state with recent history or use recurrent/attention-based architectures to approximate a fuller state. But the MDP framing is still the lens almost every RL algorithm is built around.
Value Functions and Q-Learning
A value function estimates how good it is to be in a given state (or state-action pair) under a given policy, in terms of expected future reward. Q-learning, one of the foundational RL algorithms, learns a Q-function Q(state, action) directly from experience, without ever needing an explicit model of the environment’s dynamics.
The update rule is deceptively simple: after taking an action and observing the reward and next state, adjust your current estimate of Q(state, action) toward the observed reward plus your best estimate of the value achievable from the next state. Repeated enough times, across enough state-action pairs, this converges toward the true optimal Q-function, at which point the optimal policy is just “always pick the action with the highest Q-value.”
Q-learning works well when the state space is small enough to represent as a table. Deep Q-Networks (DQN) (Mnih et al., 2013), the algorithm DeepMind used to learn to play Atari games directly from raw pixels, replaced the table with a neural network that approximates Q-values for arbitrarily large or continuous state spaces, outperforming all prior approaches on six of seven tested games and beating a human expert on three, which is what made RL viable for anything beyond toy problems.
Policy Gradients
Q-learning learns values and derives a policy indirectly. Policy gradient methods take a different approach: they parameterize the policy directly (usually as a neural network that outputs a probability distribution over actions) and adjust its parameters to directly increase the probability of actions that led to higher reward.
This matters for continuous action spaces (steering angle, joint torque) where “pick the action with the highest Q-value” isn’t well-defined the way it is with a small discrete action set. It’s also the family of algorithms behind PPO (Proximal Policy Optimization) (Schulman et al., 2017), which is the specific algorithm most commonly used in RLHF pipelines, including the ones used to fine-tune the instruction-following behavior of most current chat-oriented LLMs. PPO adds a constraint that keeps each policy update close to the previous policy, which in practice makes training dramatically more stable than earlier, more aggressive policy gradient methods.
| Family | How it picks actions | Strength | Where it shows up |
|---|---|---|---|
| Value-based (Q-learning, DQN) | Learn a value for each state-action pair, act greedily | Sample-efficient for discrete action spaces | Atari-style game agents, discrete control |
| Policy gradient (REINFORCE, PPO) | Directly learn a probability distribution over actions | Works natively with continuous action spaces | Robotics control, RLHF fine-tuning of LLMs |
| Actor-critic (A3C and descendants) | A “critic” value function reduces variance for an “actor” policy | Combines sample-efficiency of value methods with policy-gradient flexibility | Modern large-scale RL training, including many production RLHF setups |
Why RL Shows Up in LLM Training
It’s worth being specific about where RL actually sits in the LLM training pipeline, since it’s frequently mentioned in AI news without explanation. After pretraining (next-token prediction on huge text corpora) and supervised fine-tuning (training on curated example conversations), most instruction-tuned models go through an RLHF stage. This process was formalized in OpenAI’s InstructGPT work (Ouyang et al., 2022): a reward model is trained to predict which of two responses a human would prefer, and the language model is then fine-tuned with PPO to maximize that learned reward, i.e., to produce more responses that humans (or in newer pipelines, other AI judges) would rate highly. The InstructGPT paper’s own headline result was striking: human evaluators preferred outputs from a 1.3-billion-parameter InstructGPT model over outputs from the original 175-billion-parameter GPT-3, despite having over 100 times fewer parameters, purely from the RLHF fine-tuning step.
This is the step most responsible for the difference in feel between a raw pretrained base model, which will happily continue a prompt in unpredictable directions, and a released assistant model, which stays on-task and follows instructions. A newer approach, Direct Preference Optimization (DPO) (Rafailov et al., 2023), skips the explicit reward model and RL loop entirely in favor of a simpler supervised objective, directly increasing the probability of preferred responses relative to dispreferred ones, that achieves a similar effect largely because full RLHF is notoriously finicky to tune and requires sampling from the model during training. When you read that a new model was trained “with RLHF” versus “with DPO,” this is the distinction being referenced, and it’s a genuinely active research question which approach produces better-aligned models at frontier scale.
The Exploration Problem
One challenge unique to RL, and largely absent from supervised learning, is the exploration-exploitation tradeoff: an agent that only ever takes the action it currently believes is best will never discover that a different, untried action might actually be better. Balancing exploration (trying new things) against exploitation (using what you already know works) is a central design problem in every RL algorithm, handled through techniques ranging from simple random exploration (epsilon-greedy, common in DQN-style setups) to more principled approaches based on uncertainty estimation.
This is also why RL is data-hungry and comparatively fragile to train relative to supervised learning: an agent has to actually experience an outcome to learn from it, and bad exploration choices early in training can send the whole learning process down an unproductive path that’s expensive to recover from. It’s a large part of why AlphaGo’s training pipeline (Silver et al., 2016) combined supervised learning from human expert games with self-play reinforcement learning rather than relying on RL from scratch: bootstrapping from human data gave the policy a reasonable starting point before self-play RL took over to refine it further, which made the exploration problem tractable for a game as vast as Go.
Reward Hacking: When the Agent Optimizes the Wrong Thing
RL’s core promise, an agent that maximizes a reward signal, is also its sharpest failure mode. An agent will happily find a way to maximize the literal reward function it was given, even if that isn’t the outcome the designer actually intended, a phenomenon DeepMind researcher Victoria Krakovna has documented extensively in a widely-cited running list of real examples (Specification Gaming, DeepMind). Two examples from that list illustrate the point well: a boat-racing agent trained to maximize score discovered it could rack up more points by driving in tight circles through a bonus zone forever rather than actually finishing the race, since the score function rewarded that behavior even though it wasn’t what the designers meant by “play the game well.” A separate Atari-playing agent found and exploited a scoring bug that made platforms in the game blink in a way that generated essentially unlimited points, again technically maximizing the specified reward while completely defeating the intended goal of the game.
The distinction Krakovna draws is between reward gaming (the reward function itself was specified incorrectly, so a technically-optimal agent still does the wrong thing) and reward tampering (the reward signal is delivered or implemented incorrectly, and the agent exploits that implementation bug rather than the specification). Both point to the same underlying lesson: an RL agent optimizes exactly what you measure, not what you meant, which is why reward function design is treated as one of the hardest and most safety-relevant parts of building a real RL system, especially as RL techniques get applied to more open-ended domains like LLM agents operating with real tool access.
Why This Still Matters Beyond Games
RL’s reputation was built on game-playing headlines, but its actual footprint in production AI systems today is mostly invisible: it’s the fine-tuning step quietly running behind essentially every chat-oriented LLM release, the training loop behind most warehouse and manufacturing robotics, and increasingly the mechanism used to teach models multi-step tool use and agentic behavior, a topic covered in more depth in our explainer on agentic AI systems. Understanding the value-based versus policy-gradient distinction above is usually enough to follow what a new model’s technical report means when it describes its post-training pipeline.
