6.4.2
Preparing the Preference Dataset
Understand the prompt, chosen and rejected dataset structure used by DPO.
What is it?
DPO training data contains a prompt, a chosen response and a rejected response. The source uses the Argilla UltraFeedback binarized preferences dataset.
Key points
1. prompt contains the input.
2. chosen contains the preferred response.
3. rejected contains the less-preferred response.
4. Both responses relate to the same prompt.
5. The source dataset is argilla/ultrafeedback-binarized-preferences.
Preference Dataset Format
Example record:
{
"prompt": "Explain photosynthesis",
"chosen": "A clear explanation...",
"rejected": "An unclear explanation..."
}
Source dataset:
argilla/ultrafeedback-binarized-preferences
Real-world application
Prepare and validate preference pairs before passing the dataset to DPOTrainer.
Key takeaway
DPO requires correctly structured chosen and rejected response pairs.