OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / Full PPO Loss
Lesson 169 of 415 5 min read
5.3.2

Value Function Loss

Understand how the critic learns to predict target returns.

What is it?

The value function loss trains the critic to predict target returns. It measures the squared difference between the critic prediction and the target value. The source specifies a coefficient of 0.1 and notes that the value function is also clipped for stability.

Key points

1. The critic predicts the value of a state. 2. The target represents the return to be predicted. 3. Value loss measures squared prediction error. 4. Formula: L_VF = (Vθ(s_t) − V_target_t)². 5. The source specifies c1 = 0.1. 6. Value loss is also clipped for stability.

Example: Critic Prediction Error

Suppose the critic predicts a value of 6 while the target return is 8. Value loss = (6 − 8)² = 4. The squared error measures how far the critic prediction is from the target. Training adjusts the critic to reduce this error.

Real-world application

In PPO, the critic helps estimate returns and supports advantage estimation during policy training.
Key takeaway

Value function loss trains the critic to predict returns more accurately.