What is Overfitting?

Overfitting happens when a model has enough capacity to memorize the training data rather than learning generalizable patterns. The model achieves high accuracy on training data but fails on test data, because it has fit to noise rather than signal. This is the practical manifestation of the bias-variance tradeoff, where excessive model complexity leads to high variance.

Signs of overfitting include a large gap between training and validation performance, training loss continuing to decrease while validation loss increases, and model performance degrading when exposed to slightly different data distributions. Learning curves that plot training and validation metrics over time are the primary diagnostic tool.

Prevention strategies operate at multiple levels. Data strategies include collecting more training data, applying data augmentation, and using proper train/validation/test splits. Architectural strategies include reducing model size, using simpler architectures, and employing bottleneck layers. Regularization techniques include L1/L2 weight penalties, dropout (randomly deactivating neurons during training), early stopping (halting training when validation performance stops improving), and batch normalization. Ensemble methods like random forests and bagging reduce overfitting by averaging multiple models.

For deep learning, specific anti-overfitting techniques include weight decay, learning rate scheduling, gradient clipping, and label smoothing. Pre-trained models are less prone to overfitting on small datasets because they start with general features. Cross-validation provides more reliable performance estimates than single train-test splits.