OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / Motivation and History
Lesson 162 of 415 5 min read
5.1.1

The Problem with Vanilla Policy Gradients

Understand how unconstrained policy updates can cause instability and policy collapse.

What is it?

Vanilla policy gradient methods update a policy using reward-based gradients, but they do not directly limit the size of each update. A single unlucky batch can push the policy into a poor region, leading to bad outputs, lower rewards and potentially further deterioration.

Key points

Policy updates are driven by gradients; Update size is not directly constrained; Large updates can destabilize the policy; Poor outputs can lead to lower rewards; Repeated deterioration can cause policy collapse.

Illustration: An Unstable Policy Update

Imagine an AI assistant whose policy is updated using feedback from a training batch. One unusually influential batch causes a very large parameter update. The assistant begins producing poor answers and receives lower rewards. Further updates may make its performance even worse. This illustrates the instability described in the source.

Real-world application

When training an AI agent, monitor policy performance and reward trends after updates. Avoid assuming that a larger gradient update always produces better behavior.
Key takeaway

Vanilla policy gradients can make unstable updates because they do not directly constrain the update size.