Book 08 · AI & Machine LearningPage 34 of 52

Overfitting

In Turkish: Aşırı Öğrenme

Updated September 30, 20262 min read

In short

Overfitting happens when a machine learning model learns its training data so closely, including its noise, that it performs poorly on new, unseen data.

What is overfitting?

A machine learning model is supposed to learn general patterns that also hold for data it has never seen. Overfitting happens when the model instead memorizes the specific examples it was trained on, including random noise and quirks. The telltale sign is a big gap: very high accuracy on the training data but noticeably worse results on a separate validation or test set.

Overfitting is more likely when a model is very flexible compared with the amount of data, for example a large neural network trained on a few hundred examples, or when training runs for too long. Common remedies are collecting more and more varied data, using a simpler model, data augmentation, regularization techniques such as weight decay and dropout that discourage overly complex solutions, and early stopping, which halts training once validation performance stops improving.

Think of a student who memorizes the answers to last year's exam instead of understanding the subject. They ace the practice test but struggle as soon as the questions change. Overfitting is a concern in every kind of machine learning, from spam filters and price predictors to fine-tuning large language models on a small dataset.

The opposite problem is underfitting, where the model is too simple to capture the real pattern and performs poorly even on its training data. Good models sit between the two, a balance often described as the bias-variance trade-off. Overfitting is also different from data leakage, where test information accidentally sneaks into training and makes a model look better than it really is.

Key takeaways

An overfit model does well on training data but poorly on new data.
Always measure performance on held-out data the model never trained on.
More data, simpler models, regularization, and early stopping reduce overfitting.
Underfitting is the opposite: the model is too simple to learn the pattern.
Cross-validation gives a more reliable estimate of how well a model generalizes.

Example

Spotting overfitting with a train/test split (scikit-learn)python

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = make_classification(n_samples=500, n_features=20, flip_y=0.1, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

# An unlimited-depth tree can memorize the training set, noise included
deep = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
print(deep.score(X_train, y_train), deep.score(X_test, y_test))  # 1.0 vs noticeably lower

# Limiting depth forces simpler rules and usually narrows the gap
shallow = DecisionTreeClassifier(max_depth=4, random_state=0).fit(X_train, y_train)
print(shallow.score(X_train, y_train), shallow.score(X_test, y_test))

Readers ask

How do you know if a model is overfitting?

Compare its performance on the training data with its performance on a validation or test set it never saw during training. If training accuracy keeps improving while validation accuracy stalls or gets worse, the model is overfitting.

What is the difference between overfitting and underfitting?

An overfit model is too complex and learns noise, so it does well on training data but poorly on new data. An underfit model is too simple and does poorly on both.

Can large language models overfit?

Yes. Fine-tuning a large model on a small dataset for too many steps can make it repeat training examples word for word and lose general abilities, so practitioners watch validation loss and keep fine-tuning runs short.