1.5.8
Learning Rate Schedules
What is it?
A learning rate schedule defines how the learning rate changes during training. Instead of keeping it constant, a schedule can increase, decrease or vary the learning rate over time. Common approaches include linear decay, cosine decay and constant learning rates with warmup.
Key points
1. A schedule controls learning rate over training steps.
2. Linear decay gradually reduces the learning rate.
3. Cosine decay follows a cosine-shaped reduction.
4. Constant schedules keep the learning rate unchanged after any warmup.
5. Schedules can include warmup and decay phases.
6. Schedule choice depends on training duration and optimization goals.
Example: A warmup and cosine decay schedule
A training run can use three phases:
Phase 1: Warmup
The learning rate increases gradually.
Phase 2: Peak learning rate
The learning rate reaches its target value.
Phase 3: Cosine decay
The learning rate gradually decreases following a cosine curve.
This is one commonly used scheduling pattern.
Real-world application
Learning rate schedules help control optimization throughout pretraining and fine-tuning.
Key takeaway
A learning rate schedule changes update size over time to support different stages of training.