7.5.10.3
CISPO: Combining Reward Scaling and Token-Level Loss
Understand how CISPO combines batch-level reward normalisation with DAPO-style token-level loss aggregation and asymmetric clipping.
What is it?
CISPO combines batch-level reward normalisation with DAPO-style token-level loss aggregation and asymmetric clipping. Its loss also uses an overlong-filtering mask to control which completion tokens contribute.
Key points
CISPO uses batch-level reward scaling.|It adopts DAPO-style token-level loss aggregation.|It uses asymmetric clipping.|The overlong-filtering mask is represented by m(i,t).
Combining batch rewards with token-level optimisation
CISPO brings together two ideas: rewards are normalised across the batch, while the policy loss is aggregated at token level using DAPO-style asymmetric clipping. The overlong-filtering mask controls the tokens included in the loss.
Real-world application
Understand these combined components when comparing CISPO with standard GRPO and DAPO.
Key takeaway
CISPO combines batch-level reward scaling and token-level optimisation in one objective.