OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 313 of 415 5 min read
7.5.8.4

Handling Stale and Asymmetric Trajectories

Explore how VESPO down-weights stale trajectories and treats high-weight and low-weight trajectories asymmetrically.

What is it?

VESPO naturally handles stale trajectories through its exponential weighting term. Its kernel is asymmetric, treating high-weight trajectories differently from low-weight trajectories.

Key points

The kernel is smooth everywhere.|Low-weight stale trajectories are down-weighted.|The exponential term handles staleness.|High and low importance weights are treated asymmetrically.

Handling stale and high-weight trajectories

A trajectory with a very low importance weight is considered stale and is naturally down-weighted by the kernel. A trajectory with a high importance weight is treated differently, making the overall weighting asymmetric.

Real-world application

Consider VESPO for asynchronous or off-policy training where generated trajectories may become stale.
Key takeaway

VESPO is designed to handle trajectory staleness without discontinuous clipping boundaries.