1.5.3
Adam: Adaptive Moment Estimation
What is it?
Adam stands for Adaptive Moment Estimation. It combines two moving averages: the first estimates the average gradient, while the second estimates the average squared gradient. These statistics are used to adapt parameter updates. Adam also applies bias corrections to its moment estimates, especially important during early optimization steps.
Key points
1. Adam maintains a moving average of gradients.
2. This first moment estimates the gradient direction.
3. It maintains a moving average of squared gradients.
4. This second moment estimates gradient magnitude or variability.
5. Bias correction adjusts the moment estimates.
6. Updates use both moment estimates and a learning rate.
Example: How Adam adapts updates
Consider two parameters with different gradient histories.
One parameter receives consistently large gradients, while another receives smaller gradients.
Adam uses its moving averages to scale their updates differently.
This allows the optimizer to adapt update sizes based on the observed gradient history rather than relying only on one global scaling factor.
Real-world application
Adam is used in neural network training and has been an important optimizer in many deep learning workflows.
Key takeaway
Adam combines momentum-like gradient averaging with adaptive scaling based on squared gradients.