OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / PPO for RLHF: The Full Loop
Lesson 181 of 415 5 min read
5.6.2

Reward and KL Feedback

Understand how reward signals and KL constraints guide policy updates.

What is it?

After responses are generated, a reward model evaluates each prompt-response pair and assigns a score. The example uses reward scores ranging from 0.2 to 0.95. These scores represent the reward model assessment of response quality.

Key points

1. The reward model evaluates each prompt-response pair. 2. Each response receives a reward score. 3. Example score range: 0.2–0.95. 4. Higher scores indicate responses preferred by the reward model. 5. These scores are used to calculate the final training reward.

Example: Scoring Generated Responses

Suppose the reward model evaluates three responses: Response A: 0.95 Response B: 0.62 Response C: 0.20 Response A receives the highest reward-model score, while Response C receives the lowest. These scores are not yet the final PPO rewards because the KL penalty is applied in the next steps.

Real-world application

A reward model converts qualitative response preferences into numerical signals that can guide policy optimization.
Key takeaway

The reward model provides a numerical quality score for every generated response.