1.8.1
SFT Objective
What is it?
Supervised Fine-Tuning (SFT) trains a pretrained language model using curated prompt-response examples. Its objective is similar to causal language modeling, but the loss is calculated only on response tokens. Prompt tokens provide context and are excluded from the loss calculation.
Key points
1. SFT uses curated prompt-response pairs.
2. The model learns to generate the expected response.
3. Prompt tokens are used as context.
4. Loss is calculated only on response tokens.
5. Prompt labels are commonly set to -100 to exclude them from loss calculation.
How response-only training works
Prompt: What is photosynthesis?
Expected response: Photosynthesis is the process by which green plants convert light energy into chemical energy.
During SFT, the prompt provides context. The training loss is calculated on the expected response tokens rather than the prompt tokens.
Real-world application
Prepare prompt-response datasets and configure labels so that prompt tokens are masked while the intended response tokens contribute to the training loss.
Key takeaway
SFT teaches the model how to respond by learning from curated answers, not by calculating loss on the prompt itself.