6.8.1
Global Batch Size Target
Understand the recommended global batch size range for DPO and the effects of values that are too small or too large.
What is it?
DPO implementations commonly target a global batch size between 32 and 128 preference pairs. Below 32, noisy gradient estimates can cause the policy to oscillate between alignment goals such as helpfulness and safety. Above 128, convergence improvements diminish while distributed communication overhead increases.
Key points
Recommended global batch: 32–128
Below 32: Severe gradient noise in implicit reward estimation
Above 128: Diminishing convergence returns and higher communication overhead
The target is a balance between optimization stability and compute efficiency.
Choosing a Global Batch Size
Suppose a DPO training run uses a global batch size of 16. This is below the recommended range and may produce unstable policy updates. Increasing the global batch to 64 places it within the recommended range. Increasing it to 256 may add communication overhead without proportional convergence benefits.
Real-world application
Use the 32–128 range as an initial planning guide when configuring DPO training. Monitor training stability and resource utilization rather than assuming that a larger batch is always better.
Key takeaway
A global batch size between 32 and 128 is the recommended starting range for balancing DPO training stability and efficiency.