OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / Optimization Theory for LLM Training
Lesson 35 of 415 6 min read
1.5.8

Learning Rate Schedules

What is it?

A learning rate schedule defines how the learning rate changes during training. Instead of keeping it constant, a schedule can increase, decrease or vary the learning rate over time. Common approaches include linear decay, cosine decay and constant learning rates with warmup.

Key points

1. A schedule controls learning rate over training steps. 2. Linear decay gradually reduces the learning rate. 3. Cosine decay follows a cosine-shaped reduction. 4. Constant schedules keep the learning rate unchanged after any warmup. 5. Schedules can include warmup and decay phases. 6. Schedule choice depends on training duration and optimization goals.

Example: A warmup and cosine decay schedule

A training run can use three phases: Phase 1: Warmup The learning rate increases gradually. Phase 2: Peak learning rate The learning rate reaches its target value. Phase 3: Cosine decay The learning rate gradually decreases following a cosine curve. This is one commonly used scheduling pattern.

Real-world application

Learning rate schedules help control optimization throughout pretraining and fine-tuning.
Key takeaway

A learning rate schedule changes update size over time to support different stages of training.