OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / Optimization Theory for LLM Training
Lesson 29 of 415 6 min read
1.5.2

Why Vanilla SGD Fails for LLMs

What is it?

Vanilla stochastic gradient descent (SGD) updates model parameters using gradients calculated from individual training examples or mini-batches. Although simple, it can be difficult to optimize very large language models using vanilla SGD alone. Training may be sensitive to learning rates, noisy gradients and differences in parameter scales.

Key points

1. SGD uses gradients to update model parameters. 2. Mini-batch gradients can be noisy. 3. A single global learning rate may not suit every parameter. 4. Training can be sensitive to the learning rate. 5. Convergence may be slow in some optimization settings. 6. Adaptive optimizers such as Adam use additional gradient statistics.

Example: Why adaptive updates can help

Imagine two model parameters: Parameter A receives gradients that are consistently large. Parameter B receives gradients that are usually very small. Vanilla SGD applies the same global learning rate to both parameters. An adaptive optimizer can adjust updates using information from previous gradients, potentially making optimization more effective.

Real-world application

Understanding SGD limitations helps when selecting optimizers for large-scale language model pretraining and fine-tuning.
Key takeaway

Vanilla SGD is a useful foundation, but large language model training often benefits from optimizers that adapt to gradient behavior.