10.6.5
SFT Best Practices Before Reinforcement Learning
Review the source recommendations for packing, masking, chat templates, data mixing, LoRA, pass@k, training duration and diversity monitoring.
What is it?
The source summarises eight SFT practices that prepare a model for reinforcement learning: efficient sequence packing, completion-only masking, correct chat templates, balanced data mixing, LoRA, pass@k evaluation, limited training duration and diversity monitoring.
Key points
Use sequence packing to maximise GPU utilisation.
Apply completion-only masking to focus gradients on assistant responses.
Use the correct chat template for the model family.
Mix data proportionally with temperature scaling (T approximately 2) for multi-task SFT.
Use LoRA to help prevent catastrophic forgetting.
Evaluate pass@k before RL.
Avoid over-training; 1–3 epochs is usually sufficient for instruction fine-tuning.
Monitor entropy and n-gram diversity to detect mode collapse.
SFT Readiness Checklist
Before starting RL:
1. Enable sequence packing where appropriate.
2. Apply completion-only loss masking.
3. Verify the model-specific chat template.
4. Select an appropriate multi-task data mixing strategy.
5. Consider LoRA for parameter-efficient fine-tuning.
6. Measure pass@k on the target tasks.
7. Keep SFT training to an appropriate duration; the source suggests 1–3 epochs.
8. Monitor entropy and n-gram diversity.
Real-world application
Use this checklist as a pre-RL review. Record task-level pass@k, data coverage, training duration and diversity metrics so the RL starting point can be assessed consistently.
Key takeaway
The quality and diversity of the SFT model determine how much useful improvement RL can achieve.