GeneralizationCore8 minUDL ch. 8✓ ReviewedNew

Overfitting and Underfitting

A model that memorizes its training data fails on new data.

By Meta Circle Team· Builds on:Loss Functions

Step 1

Predict

A more flexible curve can bend to pass closer to the blue training points. If you keep increasing its flexibility, what happens to its distance from the orange (new) points?

Step 2

Experiment

A student memorizes all the answers in the workbook. Every workbook question: correct. A new question: lost. Models can make the same mistake; it's called overfitting.

Blue dots are the data the model learns from; orange circles are data it never sees. The degree of the polynomial controls how bendy the curve can be. Drag it from 1 to 12 and watch both errors.

Experimentlocked

🔒 Make a prediction first to unlock the experiment.

Loading experiment…

Tip: Watch the test error readout while you drag.

Step 3

Principle

The model is a degree-dd polynomial fitted by least squares on NN training points:

f(x)=w0+w1x+⋯+wdxd,L(w)=1N∑i=1N(f(xi)−yi)2.f(x) = w_0 + w_1x + \dots + w_dx^d, \qquad L(w) = \frac1N\sum_{i=1}^N\big(f(x_i)-y_i\big)^2.

Higher dd means more functions the model can represent, so the training error only goes down. Once d≥N−1d \ge N-1 the curve can pass through every training point and training error hits 0.

The test error behaves differently: it falls, then rises — a U-shape. On the left the model is too simple (high bias); on the right it has learned the noise (high variance).

Step 4

Challenge

Question 1 · lab goal

Drag the degree until the curve passes through all the blue points (training error below 0.002).

training error: — best so far
Question 2 · lab goal

Now watch the orange circles. Set the degree where the test error is (close to) its lowest.

test error ÷ best test error: — best so far
Question 3

A model has very low training error but high test error. It is most likely…

Question 4

10 points with distinct x-values. What is the lowest polynomial degree guaranteed to pass through all of them?