1.5.6
Learning Rate: The Most Important Hyperparameter
What is it?
The learning rate controls the size of parameter updates during optimization. It determines how strongly the model responds to calculated gradients. A learning rate that is too high can make training unstable, while one that is too low can make learning very slow. Selecting an appropriate learning rate is therefore a central training decision.
Key points
1. The learning rate controls update magnitude.
2. A high learning rate can cause unstable training.
3. A low learning rate can slow convergence.
4. Different optimizers can require different learning rates.
5. The best learning rate depends on the model and training setup.
6. Learning rates are often adjusted during training.
Example: Comparing learning rates
Suppose a parameter is 5.0 and its gradient is 2.0.
With learning rate 0.1:
New value = 5.0 - 0.2 = 4.8
With learning rate 0.01:
New value = 5.0 - 0.02 = 4.98
The larger learning rate produces a larger update in this simplified example.
Real-world application
Learning rate selection is important in pretraining, supervised fine-tuning and reinforcement learning optimization.
Key takeaway
The learning rate determines the scale of optimizer updates and strongly influences training stability and speed.