6.8.2
Mathematical Decomposition
Understand how global batch size is calculated from micro-batch size, number of GPUs and gradient accumulation steps, including the DPO pairing multiplier.
What is it?
DPO global batch size is determined by the per-device micro-batch size, the number of GPUs and the number of gradient accumulation steps. Each preference pair contains a prompt, a chosen response and a rejected response. The source counts the two responses as the sequence load, giving T_sequences = 2 × B_micro.
Key points
B_global = B_micro × N_GPUs × K_accum
B_micro: Preference pairs processed per device in one forward pass
N_GPUs: Number of parallel processing devices
K_accum: Gradient accumulation steps before a weight update
T_sequences = 2 × B_micro
For models above 7B parameters on 80GB GPUs with 4096–8192 token contexts, the source gives B_micro = 1–2.
Calculating the Global Batch Size
For one GPU, B_micro = 2 and K_accum = 32 gives B_global = 2 × 1 × 32 = 64. For an 8-GPU node, B_micro = 2 and K_accum = 4 gives B_global = 2 × 8 × 4 = 64. Both configurations reach the same global batch target.
Real-world application
Calculate the global batch size before training. Account for the paired chosen and rejected responses when estimating the micro-batch memory requirement.
Key takeaway
Global batch size depends on micro-batch size, GPU count and gradient accumulation; DPO processes two response sequences per preference pair.