OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / Optimization Theory for LLM Training
Lesson 31 of 415 6 min read
1.5.4

AdamW: Decoupled Weight Decay

What is it?

AdamW is an optimizer that separates weight decay from the gradient-based update used by Adam. In traditional Adam with coupled weight decay, the regularization term affects the gradient statistics. AdamW applies weight decay separately, providing a cleaner implementation of this regularization approach.

Key points

1. AdamW builds on the Adam optimizer. 2. Weight decay helps discourage excessively large parameter values. 3. Traditional coupled weight decay enters the gradient calculation. 4. AdamW decouples weight decay from the adaptive gradient update. 5. The separation changes how regularization interacts with adaptive moments. 6. Weight decay settings should be selected according to the training configuration.

Example: Understanding decoupled weight decay

Consider a model parameter updated by Adam. With coupled weight decay, the regularization contribution is included in the gradient used by the optimizer. With AdamW, the weight decay operation is applied separately from the adaptive gradient update. This distinction is important because Adam scales gradient updates using its moment estimates.

Real-world application

AdamW is widely used in Transformer training and fine-tuning configurations where decoupled weight decay is desired.
Key takeaway

AdamW separates weight decay from Adam’s adaptive gradient calculation.