Learning Rate
How big each step is. Too small never arrives; too big jumps over the valley.
Step 1
Predict
You raise the learning rate from 0.1 to 1.05. What will the ball do?
Step 2
Experiment
Gradient descent tells us which way to go; the learning rate decides how far. It is the single most commonly tuned number in training.
Pick a learning rate, then press Autoplay. Try a tiny one (0.01), a medium one (0.3–0.5), a large one (0.9) and one above 1.
🔒 Make a prediction first to unlock the experiment.
Loading experiment…
Tip: Compare how many steps each learning rate needs to reach the green dot.
Step 3
Principle
For each step satisfies , so
a geometric sequence with ratio :
- , i.e. : converges. For the ratio is negative, so the ball bounces from side to side.
- : ratio 0 — one step and done.
- : ratio −1 — bounces between two points forever.
- : diverges.
Show the math
Show the math. For a general quadratic the critical learning rate is : the steeper the valley, the smaller the usable learning rate. Real models are steep in some directions and flat in others, which is why adaptive optimizers such as Adam adjust the step per parameter.
Step 4
Challenge
Choose a learning rate that gets the ball to the bottom in 10 steps or fewer.
With a learning rate of 1.1 the ball will…
The main problem with a learning rate that is too small is…