10.6.3
Practical Implications for SFT Data and Training
Apply data quality, task coverage, training duration and task-specific warm-up principles before RL.
What is it?
The source identifies four practical requirements for preparing an SFT model for RL: high-quality and diverse data, coverage of the target tasks, appropriate training duration and a task-specific warm-up when needed.
Key points
Data quality: a small amount of high-quality data is better than a large amount of low-quality data.
Data coverage: include the tasks that RL is expected to improve.
Training duration: avoid over-training because it reduces diversity and makes exploration harder.
Warm-up: consider short task-specific SFT before RL, even when the base model is already instruction-tuned.
Preparing a Model for Mathematics RL
A model is intended for mathematics RL but its SFT dataset contains mostly general conversation examples.
A preparation plan would add high-quality mathematics examples, check coverage of the target problem types, avoid excessive SFT epochs and run a short mathematics-specific warm-up if required.
Real-world application
Audit the SFT dataset against the planned RL tasks. Evaluate quality, diversity and coverage, then choose a training duration that preserves the model’s ability to explore alternative responses.
Key takeaway
Good RL outcomes begin with high-quality, diverse SFT data that covers the target tasks without over-training.