1.5.2
Why Vanilla SGD Fails for LLMs
What is it?
Vanilla stochastic gradient descent (SGD) updates model parameters using gradients calculated from individual training examples or mini-batches. Although simple, it can be difficult to optimize very large language models using vanilla SGD alone. Training may be sensitive to learning rates, noisy gradients and differences in parameter scales.
Key points
1. SGD uses gradients to update model parameters.
2. Mini-batch gradients can be noisy.
3. A single global learning rate may not suit every parameter.
4. Training can be sensitive to the learning rate.
5. Convergence may be slow in some optimization settings.
6. Adaptive optimizers such as Adam use additional gradient statistics.
Example: Why adaptive updates can help
Imagine two model parameters:
Parameter A receives gradients that are consistently large.
Parameter B receives gradients that are usually very small.
Vanilla SGD applies the same global learning rate to both parameters.
An adaptive optimizer can adjust updates using information from previous gradients, potentially making optimization more effective.
Real-world application
Understanding SGD limitations helps when selecting optimizers for large-scale language model pretraining and fine-tuning.
Key takeaway
Vanilla SGD is a useful foundation, but large language model training often benefits from optimizers that adapt to gradient behavior.