4.3.1
Stage 1: Supervised Fine-Tuning (SFT)
Understand how high-quality demonstrations train a base model to follow instructions.
What is it?
Supervised Fine-Tuning (SFT) is the first stage of the classic RLHF pipeline. A base language model is trained on high-quality demonstrations so that it learns to follow instructions. The resulting policy is called πSFT.
Key points
1. Start with a pretrained base language model.
2. Prepare high-quality instruction-response demonstrations.
3. Train the model to imitate these demonstrations.
4. The resulting model becomes the SFT policy πSFT.
5. SFT provides a useful starting point for reinforcement learning.
6. The model learns how to respond to instructions before reward optimization.
Example: Fine-tuning a customer-support assistant
A base model receives high-quality examples of customer-support conversations.
Example instruction: How can I track my order?
Demonstration: Please visit your order history and select Track Order to view the latest delivery status.
By learning from many such demonstrations, the model develops the ability to follow instructions and produce useful responses.
The resulting instruction-following model is the SFT policy.
Real-world application
SFT is used to prepare a capable base model before applying preference-based reinforcement learning.
Key takeaway
SFT teaches the base model to follow instructions and produces the starting policy πSFT.