1.8.2
Data Quality: The LIMA Principle
What is it?
The LIMA principle emphasizes that carefully curated, high-quality examples can be more valuable than a much larger collection of noisy examples. SFT datasets should prioritize correctness, diversity, balanced response lengths and separation from evaluation benchmarks.
Key points
1. Diversity: Include question answering, summarization, code, mathematics, creative writing and multi-turn dialogue.
2. Correctness: Responses should be factually accurate and well-formatted.
3. Length balance: Include both short and long responses.
4. Decontamination: Remove overlap with evaluation benchmarks.
5. Quality and consistency matter more than raw example count.
The LIMA principle
The book cites research showing that 1,000 carefully curated examples can match models trained on 50,000 or more noisy examples.
For example, a dataset containing accurate answers across mathematics, coding, summarization and dialogue may be more useful than a much larger dataset containing inconsistent answers.
Real-world application
Review training examples for correctness, coverage, formatting and benchmark contamination before using them for fine-tuning.
Key takeaway
A smaller, carefully curated dataset can outperform a larger noisy dataset.