OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / RL Foundations for Language Models / The RLHF Pipeline
Lesson 150 of 415 5 min read
4.3.1

Stage 1: Supervised Fine-Tuning (SFT)

Understand how high-quality demonstrations train a base model to follow instructions.

What is it?

Supervised Fine-Tuning (SFT) is the first stage of the classic RLHF pipeline. A base language model is trained on high-quality demonstrations so that it learns to follow instructions. The resulting policy is called πSFT.

Key points

1. Start with a pretrained base language model. 2. Prepare high-quality instruction-response demonstrations. 3. Train the model to imitate these demonstrations. 4. The resulting model becomes the SFT policy πSFT. 5. SFT provides a useful starting point for reinforcement learning. 6. The model learns how to respond to instructions before reward optimization.

Example: Fine-tuning a customer-support assistant

A base model receives high-quality examples of customer-support conversations. Example instruction: How can I track my order? Demonstration: Please visit your order history and select Track Order to view the latest delivery status. By learning from many such demonstrations, the model develops the ability to follow instructions and produce useful responses. The resulting instruction-following model is the SFT policy.

Real-world application

SFT is used to prepare a capable base model before applying preference-based reinforcement learning.
Key takeaway

SFT teaches the base model to follow instructions and produces the starting policy πSFT.