7.5.8.1
VESPO: Principled Reward Reshaping
Understand how VESPO derives a reward-reshaping kernel from variational inference and treats policy optimisation as approximate posterior inference.
What is it?
VESPO derives a reward-reshaping kernel from variational inference. It treats policy optimisation as approximate posterior inference and produces a smooth, asymmetric weighting mechanism.
Key points
VESPO means Variational Sequence-Level Soft Policy Optimization.|It derives its kernel from variational inference.|Policy optimisation is treated as approximate posterior inference.|The resulting kernel is smooth and asymmetric.
A principled alternative to heuristic clipping
Many GRPO variants modify clipping through heuristic rules. VESPO instead derives a reward-reshaping kernel from a variational inference framework, providing a principled basis for weighting trajectories.
Real-world application
Consider VESPO when studying policy optimisation methods that derive their weighting mechanism from a variational objective.
Key takeaway
VESPO uses variational inference to derive a smooth trajectory-weighting mechanism.