1.5.9
Gradient Clipping
What is it?
Gradient clipping limits the magnitude of gradients before the optimizer updates model parameters. It is used to reduce the impact of unusually large gradients, which can destabilize training. One common method is global norm clipping, which scales gradients when their combined norm exceeds a chosen threshold.
Key points
1. Gradient clipping acts on gradients before the optimizer update.
2. It helps control unusually large gradient magnitudes.
3. Global norm clipping considers the combined gradient norm.
4. Gradients are scaled when the norm exceeds the threshold.
5. The clipping threshold is a hyperparameter.
6. Clipping does not replace appropriate learning rate and optimizer selection.
Example: Global norm clipping
Suppose the global gradient norm is 10 and the clipping threshold is 1.
The scaling factor is approximately:
1 / 10 = 0.1
The gradients are multiplied by 0.1 so their resulting global norm becomes approximately 1.
If the gradient norm is already below the threshold, clipping is not required.
Real-world application
Gradient clipping is useful when training deep neural networks and large language models where occasional large gradients can create unstable updates.
Key takeaway
Gradient clipping limits gradient magnitude to help protect training from unusually large updates.