OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / DPO Batch Size Configuration and Scaling
Lesson 245 of 415 7 min read
6.8.4

VRAM Optimization: Pre-computing Reference Log-Probabilities

Learn how pre-computing and caching reference-model log-probabilities allows the reference model to be removed from GPU memory during DPO training.

What is it?

The reference model remains unchanged during standard DPO training, so its chosen and rejected response log-probabilities can be computed before training. These values can be cached to disk, allowing the reference model to be removed from GPU memory during policy training.

Key points

Run a reference-model forward pass over the dataset before training. Cache log π_ref(y_w|x) and log π_ref(y_l|x) to disk. Remove the reference model from GPU memory. The source reports that available GPU memory doubles. B_micro can increase from 1–2 to 4–8. TRL setting: precompute_ref_log_probs=True in DPOConfig. For 70B models, the source reports approximately 140GB saved across the cluster.

Pre-computing Reference Log-Probabilities

Before DPO training begins, run the reference model over the training dataset. Store the log-probabilities for each chosen and rejected response. During training, load these cached values instead of keeping the reference model on the GPU. This frees GPU memory for larger policy micro-batches.

Real-world application

In TRL, configure DPOConfig with precompute_ref_log_probs=True when using the reference log-probability pre-computation strategy. Ensure the cached values correspond to the dataset being trained.
Key takeaway

Pre-computing reference log-probabilities can free GPU memory and increase DPO training throughput.