Detect overfitting in a small instruction dataset

Training loss can continue to fall while unseen behaviour gets worse. With a small instruction set, duplicates, template shortcuts and label inconsistencies make this especially easy to miss.

Find the validation turn

The adaptation lab uses an authored five-epoch history.

from adaptation_cases import overfit_case

result = overfit_case()
assert result["best_epoch"] == 3
assert result["validation_worsens_after"] == 3
assert round(result["final_gap"], 2) == 0.56
assert result["training_executed"] is False

Validation loss improves through epoch three, then worsens while training loss keeps falling. These are fixtures rather than a training run.

Protect the validation split

Group by source, user, document and template before splitting. Remove exact and semantic duplicates across partitions. Keep one validation set for checkpoint selection and a sealed holdout for the final decision. If you tune repeatedly against validation, record the iterations because it becomes development evidence.

Track task acceptance and critical slices alongside loss. A model can improve token likelihood while becoming worse at JSON validity, abstention or safety. Compare checkpoints on rare instructions, paraphrases, longer inputs and sources absent from training.

Reduce epochs or adapter capacity, add diverse high-quality examples, correct noisy labels and use early stopping based on predeclared criteria. More synthetic variations can amplify the same shortcut, so audit diversity instead of counting rows.

The Generative & Agentic AI course links learning curves to split integrity and paired base-model evaluation.

Exercise

Train a small permitted classifier or toy network on grouped data. Save each checkpoint, plot training and validation metrics, select without opening the holdout, then document the final independent result.

Continue learning

This article is part of the Model adaptation and multimodal tasks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Evaluate a fine-tuned model against its base model.
• Continue with Choose a quantization level using measured task quality.

Reference: Hugging Face PEFT documentation.