7.5.10.5
ScaleRL Findings and CISPO as a Large-Scale Default
Review the reported effects of batch-level scaling, token-level loss, larger batch sizes, and the source recommendation for large-scale RL.
What is it?
The source reports modest improvements from batch-level scaling or token-level loss alone, but significantly better results when both are combined. Larger batch sizes benefit more from batch-level scaling, and CISPO is recommended as a default for large-scale RL training.
Key points
Batch-level scaling alone gives modest improvement.|Token-level loss alone gives modest improvement.|Combining both produces a synergistic improvement.|Larger batches benefit more from batch-level scaling.|The source recommends CISPO for large-scale RL training.
The main findings from ScaleRL
The ScaleRL findings suggest that the two modifications should be considered together. The combined approach performs significantly better than either modification alone, with larger batches gaining more from batch-level reward scaling.
Real-world application
Consider CISPO when planning large-scale RL training, while validating performance on the target workload.
Key takeaway
The source recommends CISPO as a default for large-scale RL training.