1.8.5
Best Practices
What is it?
SFT best practices help improve training efficiency and response quality while reducing wasted computation and overfitting. The book highlights packing, NEFTune, native chat templates and appropriate epoch counts.
Key points
1. Packing: Combine short examples into sequences separated by EOS tokens to reduce padding waste.
2. NEFTune: Add uniform noise to embeddings; the book describes alpha = 5.
3. Chat templates: Use the model native chat template to preserve expected formatting.
4. Epochs: Use approximately 2–3 epochs for large datasets.
5. Small curated datasets below 10K examples may use up to 5 epochs.
6. Excessive training can cause format memorization.
Applying SFT best practices
Suppose you are fine-tuning a chat model:
1. Format conversations using the model native chat template.
2. Pack short examples to reduce padding.
3. Consider NEFTune with alpha = 5.
4. Start with 2–3 epochs for a large dataset.
5. Monitor validation performance to detect overfitting.
These are guidelines from the book, not mandatory settings for every model.
Real-world application
Before training, validate chat formatting, packing, dataset size and epoch settings. Monitor validation results rather than relying only on training loss.
Key takeaway
Good data, correct chat templates and controlled training are essential for effective SFT.