OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 300 of 415 5 min read
7.5.6.3

Positive and Negative Advantage Temperatures

Understand the separate temperature parameters for positive and non-positive advantages and their effect on policy updates.

What is it?

SAPO uses separate temperature parameters for positive and non-positive advantages. The source identifies tau-plus for positive advantages and tau-minus for negative or zero advantages.

Key points

Positive advantages use tau-plus.|Non-positive advantages use tau-minus.|The temperatures are asymmetric.|The two gates can be tuned separately.

Two temperatures for two advantage directions

The source gives tau-plus = 1.0 for positive advantages and tau-minus = 1.05 for non-positive advantages. The slightly higher negative-side temperature creates a softer gate to avoid over-suppression.

Real-world application

Tune the positive and negative temperature parameters independently when configuring SAPO.
Key takeaway

SAPO uses asymmetric temperatures to control the two advantage directions.