OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / SFT Best Practices and Techniques / Connection to RL – SFT Quality Determines RL Ceiling
Lesson 402 of 415 6 min read
10.6.2

The Exploration–Exploitation Trade-off in SFT

Understand why an SFT model must occasionally generate correct responses to provide positive reward signals for RL.

What is it?

For RL to improve a task, the SFT model must occasionally generate correct responses. These successful samples provide a non-zero reward signal. If the model never produces a correct response, there is no positive behaviour to reinforce and the learning signal can disappear.

Key points

The SFT model must occasionally produce correct responses. Successful responses provide positive reward signals. If every response is incorrect, all rewards may be zero. With zero rewards, advantages can also be zero and no useful policy gradient is produced. The source illustrates that improving a 10% mathematics success rate to 80% may be possible, while starting from 0% gives RL no progress signal.

Why a Non-Zero Success Rate Matters

Example from the source: SFT model A solves 10% of mathematics problems correctly. RL can potentially amplify this behaviour and raise performance to 80%. SFT model B solves 0% correctly. All sampled responses receive zero reward, advantages are zero and there is no gradient signal to improve correctness.

Real-world application

Measure the SFT model’s success rate on representative task prompts before RL. Ensure that sampling can produce at least some correct responses so the reward function can distinguish successful behaviour.
Key takeaway

RL needs successful behaviour to exist in the SFT model before it can reinforce that behaviour.