7.5.8.4
Handling Stale and Asymmetric Trajectories
Explore how VESPO down-weights stale trajectories and treats high-weight and low-weight trajectories asymmetrically.
What is it?
VESPO naturally handles stale trajectories through its exponential weighting term. Its kernel is asymmetric, treating high-weight trajectories differently from low-weight trajectories.
Key points
The kernel is smooth everywhere.|Low-weight stale trajectories are down-weighted.|The exponential term handles staleness.|High and low importance weights are treated asymmetrically.
Handling stale and high-weight trajectories
A trajectory with a very low importance weight is considered stale and is naturally down-weighted by the kernel. A trajectory with a high importance weight is treated differently, making the overall weighting asymmetric.
Real-world application
Consider VESPO for asynchronous or off-policy training where generated trajectories may become stale.
Key takeaway
VESPO is designed to handle trajectory staleness without discontinuous clipping boundaries.