OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / SFT Best Practices and Techniques / Connection to RL – SFT Quality Determines RL Ceiling
Lesson 401 of 415 6 min read
10.6.1

What RL Can and Cannot Learn from an SFT Model

Identify which existing behaviours reinforcement learning can amplify, suppress or refine, and which missing capabilities it cannot introduce.

What is it?

The SFT model is the starting point for reinforcement learning. RL can amplify behaviours that already exist but are weak, suppress undesirable behaviours and refine response style and format. It cannot reliably introduce capabilities that are entirely absent, recover severe catastrophic forgetting or compensate for a systematically biased reward model.

Key points

RL can amplify existing weak behaviours. RL can suppress behaviours that are present but undesirable. RL can refine response style and format. RL cannot introduce capabilities entirely absent from the SFT model. RL cannot reliably recover severe catastrophic forgetting. RL cannot compensate for a systematically biased reward model.

What RL Can Improve

Suppose an SFT model can solve some mathematics problems but often makes mistakes. RL can reinforce successful reasoning behaviours and improve the success rate. However, if the SFT model has no ability to solve a particular task, RL has no successful behaviour to amplify. A biased reward model can also direct optimisation towards the wrong behaviour.

Real-world application

Before starting RL, test whether the SFT model already demonstrates the capabilities that the reward function is intended to improve. Check reward-model quality independently.
Key takeaway

RL refines and strengthens behaviours available in the SFT starting model; it is not a substitute for missing capabilities or a reliable reward signal.