7.5.6.5
Implementing SAPO in TRL
Learn the SAPO loss configuration and temperature settings demonstrated with TRL GRPOTrainer.
What is it?
TRL provides SAPO through GRPOConfig. The source example selects the sapo loss type, sets separate positive and negative temperatures, and uses eight generations.
Key points
Set loss_type to sapo.|Set sapo_temperature_pos to 1.0.|Set sapo_temperature_neg to 1.05.|Set num_generations to 8.|Use GRPOTrainer to run training.
Configuring SAPO in TRL
from trl import GRPOConfig, GRPOTrainer
config = GRPOConfig(
loss_type="sapo",
sapo_temperature_pos=1.0,
sapo_temperature_neg=1.05,
num_generations=8,
)
trainer = GRPOTrainer(
model=model,
reward_funcs=[reward_fn],
args=config,
train_dataset=dataset,
)
Real-world application
Use the SAPO configuration when running GRPO training with the SAPO loss in a compatible TRL environment.
Key takeaway
SAPO in TRL is configured through the loss type and its two temperature parameters.