OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / Detailed Mechanics: Logits and Policy Updates
Lesson 189 of 415 6 min read
5.7.4

The PPO Weight Lifecycle

Understand the roles and lifecycle of current and old policy weights.

What is it?

PPO maintains two networks with the same architecture but different weight values. The live policy is continuously updated, while the old policy is a frozen snapshot. At the beginning of a rollout cycle, both have identical weights and the probability ratio is 1.0. After optimization, the updated live weights become the old-policy snapshot for the next cycle.

Key points

1. Live policy π_θ contains the active weights. 2. Old policy π_θold contains a frozen snapshot. 3. At rollout start, both policies have identical weights. 4. The initial probability ratio is 1.0. 5. During mini-batch training, live weights change while old weights remain frozen. 6. The ratio can deviate from 1.0, such as 1.06 or 0.94. 7. Clipping constrains the optimization objective. 8. At the end of optimization, the old snapshot is discarded. 9. The latest live weights become the old policy in the next cycle.

Example: Moving to the Next PPO Cycle

At the beginning: Live weights = θ Old weights = θ_old Both are identical, so the ratio is 1.0. During optimization: The live policy changes through gradient updates. The old policy remains frozen. At the end: The optimized live policy is copied into a fresh old-policy snapshot. The next rollout cycle begins with identical policy weights and a ratio of 1.0.

Real-world application

Understanding the weight lifecycle is important when implementing PPO training loops and managing model checkpoints.
Key takeaway

The old policy is refreshed from the optimized live policy at the start of each new cycle.