6.8.4
VRAM Optimization: Pre-computing Reference Log-Probabilities
Learn how pre-computing and caching reference-model log-probabilities allows the reference model to be removed from GPU memory during DPO training.
What is it?
The reference model remains unchanged during standard DPO training, so its chosen and rejected response log-probabilities can be computed before training. These values can be cached to disk, allowing the reference model to be removed from GPU memory during policy training.
Key points
Run a reference-model forward pass over the dataset before training.
Cache log π_ref(y_w|x) and log π_ref(y_l|x) to disk.
Remove the reference model from GPU memory.
The source reports that available GPU memory doubles.
B_micro can increase from 1–2 to 4–8.
TRL setting: precompute_ref_log_probs=True in DPOConfig.
For 70B models, the source reports approximately 140GB saved across the cluster.
Pre-computing Reference Log-Probabilities
Before DPO training begins, run the reference model over the training dataset. Store the log-probabilities for each chosen and rejected response. During training, load these cached values instead of keeping the reference model on the GPU. This frees GPU memory for larger policy micro-batches.
Real-world application
In TRL, configure DPOConfig with precompute_ref_log_probs=True when using the reference log-probability pre-computation strategy. Ensure the cached values correspond to the dataset being trained.
Key takeaway
Pre-computing reference log-probabilities can free GPU memory and increase DPO training throughput.