OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 322 of 415 6 min read
7.5.6.5

Implementing SAPO in TRL

Learn the SAPO loss configuration and temperature settings demonstrated with TRL GRPOTrainer.

What is it?

TRL provides SAPO through GRPOConfig. The source example selects the sapo loss type, sets separate positive and negative temperatures, and uses eight generations.

Key points

Set loss_type to sapo.|Set sapo_temperature_pos to 1.0.|Set sapo_temperature_neg to 1.05.|Set num_generations to 8.|Use GRPOTrainer to run training.

Configuring SAPO in TRL

from trl import GRPOConfig, GRPOTrainer config = GRPOConfig( loss_type="sapo", sapo_temperature_pos=1.0, sapo_temperature_neg=1.05, num_generations=8, ) trainer = GRPOTrainer( model=model, reward_funcs=[reward_fn], args=config, train_dataset=dataset, )

Real-world application

Use the SAPO configuration when running GRPO training with the SAPO loss in a compatible TRL environment.
Key takeaway

SAPO in TRL is configured through the loss type and its two temperature parameters.