6.6.5
Preference Data Diversity
Understand why preference pairs should cover different quality levels and approaches rather than only good-versus-terrible responses.
What is it?
Preference pairs should cover a broad spectrum of quality differences and response approaches. Training only on obviously good-versus-terrible pairs provides less diverse policy distinctions.
Key points
1. Cover the full spectrum of response quality differences.
2. Do not rely only on good-versus-terrible comparisons.
3. Include pairs that differ in approach.
4. Different approaches teach richer policy distinctions.
5. Dataset diversity improves the range of preferences represented.
Comparing Different Response Approaches
Prompt: Explain a complex topic to a student.
Response A: A concise explanation with a simple analogy.
Response B: A detailed explanation with a step-by-step derivation.
Both responses may be useful, but they represent different approaches. Preference comparisons between them can teach more nuanced distinctions than comparing a good response with a completely incorrect one.
Real-world application
Build preference datasets containing different response styles, levels of detail and quality differences relevant to the intended use case.
Key takeaway
Diverse preference pairs teach richer distinctions than only obvious good-versus-bad comparisons.