OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 326 of 415 5 min read
7.5.10.5

ScaleRL Findings and CISPO as a Large-Scale Default

Review the reported effects of batch-level scaling, token-level loss, larger batch sizes, and the source recommendation for large-scale RL.

What is it?

The source reports modest improvements from batch-level scaling or token-level loss alone, but significantly better results when both are combined. Larger batch sizes benefit more from batch-level scaling, and CISPO is recommended as a default for large-scale RL training.

Key points

Batch-level scaling alone gives modest improvement.|Token-level loss alone gives modest improvement.|Combining both produces a synergistic improvement.|Larger batches benefit more from batch-level scaling.|The source recommends CISPO for large-scale RL training.

The main findings from ScaleRL

The ScaleRL findings suggest that the two modifications should be considered together. The combined approach performs significantly better than either modification alone, with larger batches gaining more from batch-level reward scaling.

Real-world application

Consider CISPO when planning large-scale RL training, while validating performance on the target workload.
Key takeaway

The source recommends CISPO as a default for large-scale RL training.