1.7.6
Mid-Training: Preparing for Reinforcement Learning
What is it?
Mid-training is an additional stage between raw pretraining and post-training. It continues training on curated data, often emphasizing STEM, mathematics and code, while preparing the model for later reinforcement learning.
Key points
1. Mid-training is also called continued pretraining or annealing.
2. It increases the representation of high-quality STEM, mathematics and code data.
3. It can extend context length to 128K–256K tokens.
4. It aims to develop behaviours such as verification, backtracking and subgoal decomposition.
5. Reinforcement learning can later reinforce these behaviours.
6. The book describes typical mid-training runs using 100B–500B curated tokens and a cosine-decayed learning rate.
Preparing a model for reasoning-focused training
A base model receives additional training on curated mathematics, science and code data.
This stage can encourage behaviours such as checking intermediate results, correcting mistakes and breaking complex problems into smaller steps.
A later reinforcement-learning stage can then reinforce these behaviours.
Real-world application
Use mid-training when a pretrained base model needs stronger domain coverage, longer context handling or reasoning behaviours before reinforcement-learning training.
Key takeaway
Mid-training prepares the foundation. Reinforcement learning can amplify behaviours that the model has already begun to demonstrate.