1.11.2
Training Data Diversity
What is it?
Training data diversity means including prompts and examples from different domains, difficulty levels and formats. A diverse dataset helps a model learn broader patterns instead of overfitting to a narrow collection of examples.
Key points
Prompt diversity should cover different domains, difficulty levels and formats. The source describes a Goldilocks principle in which prompts have approximately 20–80% success rate. Deduplication removes exact and near-duplicate examples using approaches such as MinHash and n-gram overlap. Data mixing balances tasks and domains through temperature-weighted sampling or curriculum strategies.
Building a Diverse Training Dataset
Imagine a training dataset containing mostly simple mathematics questions. The model may perform well on those examples but struggle with coding, reasoning and other domains. Adding varied tasks, difficulty levels and formats improves coverage. Removing near-duplicates also reduces overfitting to repeated patterns.
Real-world application
Use dataset analysis, deduplication and balanced sampling when preparing supervised fine-tuning or reinforcement-learning prompt collections.
Key takeaway
Training data should be varied, balanced and sufficiently challenging to support broad model behaviour.