7.5.8.5
Implementing VESPO in TRL
Learn the VESPO loss configuration, positive-advantage parameters, generation count, and off-policy training setting shown in TRL.
What is it?
TRL demonstrates VESPO through GRPOConfig and GRPOTrainer. The example selects the vespo loss, configures positive-advantage sharpness and staleness parameters, and uses multiple generation steps.
Key points
Set loss_type to vespo.|Set vespo_k_pos to 2.0.|Set vespo_lambda_pos to 3.0.|Use num_generations = 8.|The example uses steps_per_generation = 2.
Configuring VESPO in TRL
from trl import GRPOConfig, GRPOTrainer
config = GRPOConfig(
loss_type="vespo",
vespo_k_pos=2.0,
vespo_lambda_pos=3.0,
num_generations=8,
steps_per_generation=2,
)
trainer = GRPOTrainer(
model=model,
reward_funcs=[reward_fn],
args=config,
train_dataset=dataset,
)
Real-world application
Use this configuration as the source example when experimenting with VESPO in a compatible TRL environment.
Key takeaway
VESPO in TRL uses its own loss type and parameters for positive-advantage trajectory weighting.