7.5.10.1
ScaleRL: Scaling Laws for Reinforcement Learning
Understand the ScaleRL study and its findings about combining batch-level reward scaling with token-level loss aggregation.
What is it?
ScaleRL studies how reinforcement learning for language models scales. Its key finding is that batch-level reward scaling and DAPO-style token-level loss work synergistically: neither modification alone produces the same improvement as combining both.
Key points
ScaleRL studies scaling behaviour in LLM reinforcement learning.|It combines batch-level reward scaling with token-level loss.|Neither modification alone is sufficient for the reported strong performance.|CISPO is the resulting algorithm.
Why two training modifications work better together
The source reports that batch-level reward scaling alone gives modest improvement, and token-level loss alone also gives modest improvement. Combining both produces a significantly stronger result.
Real-world application
Use this finding when evaluating design choices for large-scale reinforcement learning training.
Key takeaway
ScaleRL identifies a synergistic benefit from combining batch-level reward scaling and token-level loss.