Gradient Descent
Downhill blindfolded: feel the slope under your feet and take a small step down.
Step 1
Predict
The learning rate stays fixed. As the ball gets closer to the bottom of the valley, how big is each step?
Step 2
Experiment
Imagine standing blindfolded on a hillside, trying to reach the valley floor. You can't see, but you can feel which way the ground tilts. So you take a small step downhill, feel again, step again.
Training a model works the same way. The curve is the loss as a function of a parameter ; the ball is the current parameter. The orange line is the tangent — its slope is the gradient. Press Take a step a few times, then try Autoplay.
🔒 Make a prediction first to unlock the experiment.
Loading experiment…
Tip: Watch how far the ball moves on each click as it approaches the minimum.
Step 3
Principle
At each step the parameter moves against the gradient, by the gradient times a small positive number , the learning rate:
Here the loss is with its minimum at , and . Substituting:
Each step multiplies the distance to the minimum by . With that's 0.8: distances shrink 0.8, 0.64, 0.512… — a geometric sequence.
With many parameters the derivative becomes the gradient — the vector of partial derivatives — and the update looks exactly the same. Real networks have millions to billions of parameters, but every step is still: compute the slope, step a little downhill. Computing that gradient efficiently is backpropagation.
Step 4
Challenge
Use Take a step or Autoplay to bring the ball within 0.01 of the minimum.
The ball is at w = 3 and the slope there goes up to the right (positive gradient). After the next step, w will be…
With a fixed learning rate, as the ball nears the bottom each step gets…
L(w) = (w − 1)². What is the derivative L′(w) at w = 3?