OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / RL Foundations for Language Models / The RLHF Pipeline
Lesson 152 of 415 7 min read
4.3.3

Stage 3: RL Optimization with PPO or GRPO

Understand how PPO or GRPO optimizes the policy against reward signals with a KL constraint.

What is it?

RL Optimization is the third stage. The reward model provides a training signal to optimize the language-model policy using PPO or GRPO. A KL constraint against the SFT policy helps prevent the updated model from drifting too far from its instruction-following starting point.

Key points

1. Use the reward model to score generated responses. 2. Optimize the language-model policy to increase reward. 3. PPO and GRPO are the optimization methods identified by the source. 4. Apply a KL constraint against πSFT. 5. The KL constraint limits policy drift. 6. The objective is to improve response quality while retaining the useful behaviour learned during SFT.

Example: Improving a trained assistant

An SFT model generates a response to a customer query. The reward model scores the response based on learned human preferences. PPO or GRPO uses this reward signal to update the model policy. The KL constraint keeps the updated policy from moving too far away from the original SFT policy. The process is repeated to improve the model.

Real-world application

This stage is the core reinforcement-learning optimization step in a classic RLHF training pipeline.
Key takeaway

PPO or GRPO optimizes the policy against reward feedback while using a KL constraint against πSFT.