ai/overfitting-why-a-perfect-score-on-practice-questions-can-still-fail.md

Overfitting: why a perfect score on practice questions can still fail

AI · 6 min read ·

A model that scores perfectly on the examples it learned from can still fail badly on new ones. That failure is called overfitting, and you can watch it happen here.

Fit a curve through ten noisy points, slide its flexibility from a straight line up to degree 9, and drag the points around. Compare the error on the training points with the error on fresh test points, and see why the second number is the one that matters.

A dark chart: ten green training points, a dashed smooth wave, and an orange curve that hits every point but swings wildly between them.

$ ls ai/

see all →
ai/tokens-why-a-chatbot-never-reads-your-letters-only-numbered-pieces.md Tokens: why a chatbot never reads your letters, only numbered pieces A chatbot never reads your letters. Before it sees anything, your text is chopped into pieces called tokens, and every piece is swapped for… 6 min read · 6 Oct ai/embeddings-how-a-computer-measures-that-a-kitten-is-like-a-cat.md Embeddings: how a computer measures that a kitten is like a cat How does a computer know that kitten is closer to cat than to car? It can't read meaning, but it can measure it. An embedding turns a word… 4 min read · 3 Oct ai/machine-learning-by-example-how-a-computer-learns-apples-from-oranges.md Machine learning by example: how a computer learns apples from oranges Normal programs follow rules a person wrote. Machine learning turns that around: you give the computer examples with the right answers, and… 11 min read · 1 Oct ai/how-a-chatbot-writes-one-guessed-word-at-a-time.md How a chatbot writes: one guessed word at a time A chatbot doesn't plan its reply and then type it out. It repeats one small step: look at the text so far, give every possible next token… 12 min read · 1 Oct