OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / Optimization Theory for LLM Training
Lesson 36 of 415 6 min read
1.5.9

Gradient Clipping

What is it?

Gradient clipping limits the magnitude of gradients before the optimizer updates model parameters. It is used to reduce the impact of unusually large gradients, which can destabilize training. One common method is global norm clipping, which scales gradients when their combined norm exceeds a chosen threshold.

Key points

1. Gradient clipping acts on gradients before the optimizer update. 2. It helps control unusually large gradient magnitudes. 3. Global norm clipping considers the combined gradient norm. 4. Gradients are scaled when the norm exceeds the threshold. 5. The clipping threshold is a hyperparameter. 6. Clipping does not replace appropriate learning rate and optimizer selection.

Example: Global norm clipping

Suppose the global gradient norm is 10 and the clipping threshold is 1. The scaling factor is approximately: 1 / 10 = 0.1 The gradients are multiplied by 0.1 so their resulting global norm becomes approximately 1. If the gradient norm is already below the threshold, clipping is not required.

Real-world application

Gradient clipping is useful when training deep neural networks and large language models where occasional large gradients can create unstable updates.
Key takeaway

Gradient clipping limits gradient magnitude to help protect training from unusually large updates.