5.9.6
Batch Size (batch_size)
Understand the typical PPO batch size range and the relationship between batch size, gradient smoothness and generation compute.
What is it?
Batch size determines how many prompts are processed in a PPO rollout batch. Larger batches generally produce smoother gradients but require more generation compute.
Key points
1. Typical range: 64–256.
2. The parameter is batch_size.
3. Larger batches can smooth gradient estimates.
4. Larger batches require more response generation.
5. Batch size affects memory, compute requirements and training efficiency.
Example: PPO Batch Size
batch_size = 128
The trainer collects 128 prompts in a rollout batch.
A larger batch can provide smoother training signals, but increases generation cost.
Real-world application
Choose batch size according to GPU memory, generation throughput and the stability of gradient estimates.
Key takeaway
Batch size balances gradient smoothness against generation compute.