OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 304 of 415 6 min read
7.5.10.3

CISPO: Combining Reward Scaling and Token-Level Loss

Understand how CISPO combines batch-level reward normalisation with DAPO-style token-level loss aggregation and asymmetric clipping.

What is it?

CISPO combines batch-level reward normalisation with DAPO-style token-level loss aggregation and asymmetric clipping. Its loss also uses an overlong-filtering mask to control which completion tokens contribute.

Key points

CISPO uses batch-level reward scaling.|It adopts DAPO-style token-level loss aggregation.|It uses asymmetric clipping.|The overlong-filtering mask is represented by m(i,t).

Combining batch rewards with token-level optimisation

CISPO brings together two ideas: rewards are normalised across the batch, while the policy loss is aggregated at token level using DAPO-style asymmetric clipping. The overlong-filtering mask controls the tokens included in the loss.

Real-world application

Understand these combined components when comparing CISPO with standard GRPO and DAPO.
Key takeaway

CISPO combines batch-level reward scaling and token-level optimisation in one objective.