7.5.8.3
The VESPO Gradient Weighting Kernel
Understand the smooth weighting function and the roles of the sharpness parameter k and staleness parameter lambda.
What is it?
VESPO transforms the trajectory importance weight into a gradient weight using a smooth kernel. The parameter k controls weighting sharpness, while lambda controls exponential decay for stale, low-weight trajectories.
Key points
The kernel is smooth.|k controls the sharpness of the weighting.|lambda controls exponential decay.|The kernel transforms W(tau) into the final gradient weight.
The roles of k and lambda
The source gives the gradient weight as g(tau) = W(tau)^k × exp(lambda(1 − W(tau))). The exponent k controls sharpness, while lambda controls how strongly stale trajectories are down-weighted.
Real-world application
Understand the two parameters before tuning VESPO in an off-policy training configuration.
Key takeaway
k controls sharpness; lambda controls staleness decay.