OptimizationCore7 minUDL ch. 6✓ ReviewedNew

Learning Rate

How big each step is. Too small never arrives; too big jumps over the valley.

By Meta Circle Team· Builds on:Gradient Descent

Step 1

Predict

You raise the learning rate from 0.1 to 1.05. What will the ball do?

Step 2

Experiment

Gradient descent tells us which way to go; the learning rate decides how far. It is the single most commonly tuned number in training.

Pick a learning rate, then press Autoplay. Try a tiny one (0.01), a medium one (0.3–0.5), a large one (0.9) and one above 1.

Experimentlocked

🔒 Make a prediction first to unlock the experiment.

Loading experiment…

Tip: Compare how many steps each learning rate needs to reach the green dot.

Step 3

Principle

For L(w)=(w−1)2L(w) = (w-1)^2 each step satisfies wt+1−1=(1−2η)(wt−1)w_{t+1}-1 = (1-2\eta)(w_t-1), so

wt−1=(1−2η)t (w0−1),w_t - 1 = (1-2\eta)^t\,(w_0-1),

a geometric sequence with ratio 1−2η1-2\eta:

  • ∣1−2η∣<1|1-2\eta| < 1, i.e. 0<η<10<\eta<1: converges. For η>0.5\eta > 0.5 the ratio is negative, so the ball bounces from side to side.
  • η=0.5\eta = 0.5: ratio 0 — one step and done.
  • η=1\eta = 1: ratio −1 — bounces between two points forever.
  • η>1\eta > 1: diverges.
Show the math

Show the math. For a general quadratic L(w)=a(w−w∗)2L(w) = a(w-w^*)^2 the critical learning rate is 1/a1/a: the steeper the valley, the smaller the usable learning rate. Real models are steep in some directions and flat in others, which is why adaptive optimizers such as Adam adjust the step per parameter.

Step 4

Challenge

Question 1 · lab goal

Choose a learning rate that gets the ball to the bottom in 10 steps or fewer.

steps needed: — best so far
Question 2

With a learning rate of 1.1 the ball will…

Question 3

The main problem with a learning rate that is too small is…