4.4.6
SFT Best Practices
Explore sequence packing, chat templates, data mixing and SFT quality.
What is it?
Supervised Fine-Tuning best practices improve the quality of the starting policy before reinforcement learning. The chapter focuses on sequence packing, chat templates, data mixing and the importance of SFT quality in determining the RL performance ceiling.
Key points
1. Sequence packing improves training-data utilization.
2. Chat templates ensure consistent conversation formatting.
3. Data mixing combines appropriate training examples.
4. Data quality influences the starting model capability.
5. SFT quality determines the RL ceiling.
6. Reinforcement learning cannot automatically compensate for every weakness in the initial SFT model.
Example: Preparing an SFT dataset
A team prepares instruction-following examples for an assistant.
Sequence packing combines shorter training sequences efficiently.
Chat templates format system, user and assistant messages consistently.
Data mixing balances different task types and capabilities.
A stronger and better-prepared SFT model gives subsequent RL training a better starting point.
Real-world application
These practices are useful when preparing instruction-tuning datasets and building reliable starting checkpoints for RLHF or RLVR.
Key takeaway
Good SFT data, formatting and training practices establish the quality ceiling for later reinforcement learning.