OptimizationIntro8 minUDL ch. 6✓ ReviewedNew

Gradient Descent

Downhill blindfolded: feel the slope under your feet and take a small step down.

By Meta Circle Team· Builds on:Loss Functions

Step 1

Predict

The learning rate stays fixed. As the ball gets closer to the bottom of the valley, how big is each step?

Step 2

Experiment

Imagine standing blindfolded on a hillside, trying to reach the valley floor. You can't see, but you can feel which way the ground tilts. So you take a small step downhill, feel again, step again.

Training a model works the same way. The curve is the loss as a function of a parameter ww; the ball is the current parameter. The orange line is the tangent — its slope is the gradient. Press Take a step a few times, then try Autoplay.

Experimentlocked

🔒 Make a prediction first to unlock the experiment.

Loading experiment…

Tip: Watch how far the ball moves on each click as it approaches the minimum.

Step 3

Principle

At each step the parameter moves against the gradient, by the gradient times a small positive number η\eta, the learning rate:

wt+1=wt−η L′(wt)w_{t+1} = w_t - \eta\, L'(w_t)

Here the loss is L(w)=(w−1)2L(w) = (w-1)^2 with its minimum at w∗=1w^*=1, and L′(w)=2(w−1)L'(w) = 2(w-1). Substituting:

wt+1−1=(1−2η)(wt−1)w_{t+1} - 1 = (1-2\eta)(w_t - 1)

Each step multiplies the distance to the minimum by ∣1−2η∣|1-2\eta|. With η=0.1\eta = 0.1 that's 0.8: distances shrink 0.8, 0.64, 0.512… — a geometric sequence.

With many parameters the derivative becomes the gradient — the vector of partial derivatives — and the update looks exactly the same. Real networks have millions to billions of parameters, but every step is still: compute the slope, step a little downhill. Computing that gradient efficiently is backpropagation.

Step 4

Challenge

Question 1 · lab goal

Use Take a step or Autoplay to bring the ball within 0.01 of the minimum.

distance to minimum: — best so far
Question 2

The ball is at w = 3 and the slope there goes up to the right (positive gradient). After the next step, w will be…

Question 3

With a fixed learning rate, as the ball nears the bottom each step gets…

Question 4

L(w) = (w − 1)². What is the derivative L′(w) at w = 3?