OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / Supervised Fine-Tuning (SFT)
Lesson 52 of 415 6 min read
1.8.2

Data Quality: The LIMA Principle

What is it?

The LIMA principle emphasizes that carefully curated, high-quality examples can be more valuable than a much larger collection of noisy examples. SFT datasets should prioritize correctness, diversity, balanced response lengths and separation from evaluation benchmarks.

Key points

1. Diversity: Include question answering, summarization, code, mathematics, creative writing and multi-turn dialogue. 2. Correctness: Responses should be factually accurate and well-formatted. 3. Length balance: Include both short and long responses. 4. Decontamination: Remove overlap with evaluation benchmarks. 5. Quality and consistency matter more than raw example count.

The LIMA principle

The book cites research showing that 1,000 carefully curated examples can match models trained on 50,000 or more noisy examples. For example, a dataset containing accurate answers across mathematics, coding, summarization and dialogue may be more useful than a much larger dataset containing inconsistent answers.

Real-world application

Review training examples for correctness, coverage, formatting and benchmark contamination before using them for fine-tuning.
Key takeaway

A smaller, carefully curated dataset can outperform a larger noisy dataset.