GeneralizationCore7 minUDL ch. 9✓ ReviewedNew

L2 Regularization

Tame an overfitting model by penalizing large weights.

By Meta Circle Team· Builds on:Overfitting and Underfitting

Step 1

Predict

A degree-12 polynomial wildly overfits 12 points. We add a penalty for large coefficients and slowly increase its strength λ. What happens to the test error?

Step 2

Experiment

Instead of choosing a smaller model, we can keep the flexible one and discourage it from using extreme weights. That's L2 regularization, also called weight decay or ridge regression.

The lab fits a degree-12 polynomial to 12 points — way too flexible. Turn up λ step by step and watch the curve and the test error.

Experimentlocked

🔒 Make a prediction first to unlock the experiment.

Loading experiment…

Tip: Try every λ value and note where the test error is smallest.

Step 3

Principle

L2 regularization adds the squared size of the weights to the loss:

Lreg(w)=1N∑i(f(xi)−yi)2+λ∑j≥1wj2.L_{\text{reg}}(w) = \frac1N\sum_i\big(f(x_i)-y_i\big)^2 + \lambda\sum_{j\ge1} w_j^2.

Large coefficients are what let a polynomial zig-zag between points; making them expensive forces a smoother curve. λ\lambda trades off fitting the training data against keeping weights small:

  • λ=0\lambda = 0: plain least squares — overfits.
  • moderate λ\lambda: smooth curve, lowest test error.
  • large λ\lambda: weights squashed toward 0 — underfits.

Step 4

Challenge

Question 1 · lab goal

Choose λ so that the test error drops below 0.15.

test error: — best so far
Question 2

What does L2 regularization add to the loss?

Question 3

With a very large λ, the model will most likely…