OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 291 of 415 5 min read
7.5.8.2

Sequence-Level Importance Weighting

Learn how VESPO calculates trajectory weights using the sequence-level importance ratio between the current and old policies.

What is it?

VESPO assigns each trajectory a sequence-level importance weight W(tau), defined as the ratio between the current policy probability and the old policy probability for that trajectory.

Key points

W(tau) is the sequence-level importance weight.|W(tau) = πθ(tau) / πold(tau).|The weight compares current and old policy probabilities.|The weighting is applied at trajectory level.

Calculating a trajectory importance weight

Suppose the current policy assigns a trajectory probability of 0.04 and the old policy assigns it 0.02. The sequence-level importance weight is 0.04 / 0.02 = 2. This illustrates the ratio used by VESPO.

Real-world application

Use the sequence-level importance ratio to understand how VESPO evaluates trajectories generated by an older policy.
Key takeaway

VESPO uses a sequence-level ratio to compare current and old policy probabilities.