7.5.8.2
Sequence-Level Importance Weighting
Learn how VESPO calculates trajectory weights using the sequence-level importance ratio between the current and old policies.
What is it?
VESPO assigns each trajectory a sequence-level importance weight W(tau), defined as the ratio between the current policy probability and the old policy probability for that trajectory.
Key points
W(tau) is the sequence-level importance weight.|W(tau) = πθ(tau) / πold(tau).|The weight compares current and old policy probabilities.|The weighting is applied at trajectory level.
Calculating a trajectory importance weight
Suppose the current policy assigns a trajectory probability of 0.04 and the old policy assigns it 0.02. The sequence-level importance weight is 0.04 / 0.02 = 2. This illustrates the ratio used by VESPO.
Real-world application
Use the sequence-level importance ratio to understand how VESPO evaluates trajectories generated by an older policy.
Key takeaway
VESPO uses a sequence-level ratio to compare current and old policy probabilities.