OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / Optimization Theory for LLM Training
Lesson 33 of 415 6 min read
1.5.6

Learning Rate: The Most Important Hyperparameter

What is it?

The learning rate controls the size of parameter updates during optimization. It determines how strongly the model responds to calculated gradients. A learning rate that is too high can make training unstable, while one that is too low can make learning very slow. Selecting an appropriate learning rate is therefore a central training decision.

Key points

1. The learning rate controls update magnitude. 2. A high learning rate can cause unstable training. 3. A low learning rate can slow convergence. 4. Different optimizers can require different learning rates. 5. The best learning rate depends on the model and training setup. 6. Learning rates are often adjusted during training.

Example: Comparing learning rates

Suppose a parameter is 5.0 and its gradient is 2.0. With learning rate 0.1: New value = 5.0 - 0.2 = 4.8 With learning rate 0.01: New value = 5.0 - 0.02 = 4.98 The larger learning rate produces a larger update in this simplified example.

Real-world application

Learning rate selection is important in pretraining, supervised fine-tuning and reinforcement learning optimization.
Key takeaway

The learning rate determines the scale of optimizer updates and strongly influences training stability and speed.