If you keep asking what is reinforcement learning, here is the short answer: a program learns by trial and error, steered by rewards. You’ll see the loop, the main methods and the limits.
If you are new to the wider field, start with what is artificial intelligence, then come back here.
| Item | Value |
|---|---|
| Typical discount factor (γ) | 0.99 |
| DQN on Atari games (superhuman) | 29 of 49 games |
| AlphaGo against Lee Sedol (2016) | Won 4-1 |
| AlphaGo hardware for that match | 1,920 CPUs and 280 GPUs |
| OpenAI Five training experience | Over 45,000 years of simulated gameplay |
These figures come from the Ch 12 reinforcement learning guide and the RL AI interactive chart. Figures can change, so check the source before you rely on them.
How does reinforcement learning work?
Reinforcement learning (RL) is the third paradigm of machine learning, alongside supervised and unsupervised learning, according to the Ch 12 guide. Instead of labelled examples, an agent takes actions in an environment and receives rewards.
No teacher supplies the correct answers. The agent builds a policy, a strategy that maximises cumulative reward over time, through trial and error.
The cycle repeats again and again:
- The agent observes the current state of the environment, such as a board position or sensor readings.
- It picks an action using its policy.
- The environment moves to a new state and returns a reward.
- The agent updates its policy to do better next time.
The reward is a single number, and the RL AI chart describes it as the training signal. Everything the agent learns comes from that number, so the way it is designed matters a great deal.
Which settings shape how an agent learns?
A few dials control the behaviour of most RL systems. The RL AI chart lists them as key parameters.
- Learning rate: how much the agent updates its estimates on each step. Fast learning can be unstable, slow learning is steadier.
- Discount factor: how much future rewards count against immediate ones. A value near 1 means long-term thinking, and a value near 0 means short-sighted.
- Exploration rate: the chance of taking a random action to discover new strategies.
- Reward design: the same chart calls it the single most critical design choice in RL.
The Ch 12 guide gives a typical discount factor of 0.99. That means rewards that arrive later are worth only slightly less than immediate ones.
Why is exploring versus exploiting hard?
An agent must choose between trying something new (exploration) and sticking with what already works (exploitation). Lean too far either way and learning suffers.
A common fix is ε-greedy. With probability ε the agent takes a random action, otherwise it takes the best known one, and ε usually starts high and shrinks as the agent learns, per the Ch 12 guide.
The same guide names another hurdle, credit assignment. If a goal is scored many moves later, which earlier move deserves the credit? Chess makes this harsh, because the reward only arrives at the end of the game.
What are the main families of methods?
Most methods fall into two camps, and many modern systems blend them.
Value-based methods
Q-learning estimates the total future reward of taking an action in a state. It updates that estimate after each experience using the Bellman equation, and once learned, the best policy is to pick the action with the highest value.
A Q-table breaks down when the state space is huge, such as raw game screens. DeepMind’s DQN, described by Mnih et al. (2013/2015), replaced the table with a neural network that learns directly from raw game pixels. The Ch 12 guide says it reached superhuman play on 29 of 49 Atari games.
Policy-based methods
Policy gradient methods skip the value table and optimise the policy itself. They raise the probability of actions that led to high rewards and lower it for those that did not.
PPO (Schulman, 2017) clips each update so the policy cannot change too much at once. The guide notes that DQN handles discrete actions only, while policy methods suit continuous control such as robotics.
Where does reinforcement learning show up?
Games gave RL its most famous results. The RL AI chart says AlphaGo’s 2016 victory over Lee Sedol used 1,920 CPUs and 280 GPUs, and OpenAI Five gathered over 45,000 years of simulated gameplay in training.
The Ch 12 guide reports AlphaGo beat Lee Sedol 4-1. Its successor AlphaGo Zero (2017) used no human data at all, learning through pure self-play.
The chart also notes that MuZero learned Go, chess, shogi and Atari games without being told the rules of any game. Robotics teams often train in simulation and then transfer the result to a real machine, known as sim-to-real transfer.
Language models and RLHF
RL also helps shape chat assistants. In reinforcement learning from human feedback (RLHF), the language model is the agent, the prompt is the state, the generated response is the action and a reward model’s score is the reward.
The guide says InstructGPT showed a 1.3B RLHF model was preferred over a 175B base model. If you want the broader picture of such systems, read what is generative ai.
What are the limits of reinforcement learning?
RL is data hungry. The Ch 12 guide says it needs millions of interactions, which makes it much less sample efficient than supervised learning.
Sparse rewards, credit assignment and careful reward design add to the difficulty. A badly chosen reward steers the agent toward the wrong behaviour.
Fitting too tightly to training conditions is a separate risk in all of machine learning. For the supervised version of that problem, see what is overfitting in machine learning.
Frequently asked questions
How is reinforcement learning different from supervised learning?
Show answer
Supervised learning uses labelled examples. Reinforcement learning has no teacher giving correct answers, so the agent learns from rewards through trial and error.
What is a policy?
Show answer
A policy is the agent’s strategy for choosing an action in a given state. Learning means improving it so total reward grows over time.
What does the discount factor do?
Show answer
It sets how much future rewards count compared with immediate ones. A value near 1 favours long-term gains, and a value near 0 favours quick wins.
Does ChatGPT-style training use reinforcement learning?
Show answer
The Ch 12 guide describes RLHF as the bridge between RL and language models, where a reward model scores responses. It also says newer methods such as DPO remove the RL step.
