1.5.7
Learning Rate Warmup
What is it?
Learning rate warmup is a training strategy in which the learning rate begins at a low value and gradually increases during an initial training period. This helps avoid applying large parameter updates before the model and optimizer have established useful training statistics.
Key points
1. Warmup starts with a relatively small learning rate.
2. The learning rate increases over a specified number of steps.
3. Warmup can use a linear or other increasing schedule.
4. It is commonly used in Transformer training configurations.
5. Warmup duration is a hyperparameter.
6. The warmup phase is usually followed by another learning rate schedule.
Example: Linear warmup
Suppose the target learning rate is 0.001 and warmup lasts 1,000 steps.
At step 0: 0.000000
At step 250: 0.000250
At step 500: 0.000500
At step 750: 0.000750
At step 1,000: 0.001000
This illustrates a linear warmup schedule.
Real-world application
Warmup is useful in large-scale model training and fine-tuning configurations where early optimization stability is important.
Key takeaway
Warmup gradually increases the learning rate at the beginning of training instead of applying the full target learning rate immediately.