1.3.9
Layer Normalization
What is it?
Layer normalization is a technique used to normalize activations within a neural network. In Transformers, it helps maintain stable numerical behavior as information passes through many layers. Different Transformer implementations use different normalization arrangements, including pre-normalization and post-normalization.
Key points
1. Layer normalization operates on activation values.
2. It normalizes values across the feature dimensions of a token representation.
3. Learned scale and bias parameters are commonly used.
4. It can improve training stability.
5. Pre-norm applies normalization before a sublayer.
6. Post-norm applies normalization after a sublayer and its residual connection.
Example: Normalization in a Transformer
A Transformer layer may follow a pre-normalization arrangement:
Input
↓
Layer Normalization
↓
Attention
↓
Residual Connection
↓
Layer Normalization
↓
Feed-Forward Network
The exact order depends on the architecture.
Real-world application
Normalization is important when training and operating deep neural networks, including large language models.
Key takeaway
Layer normalization helps stabilize the activations flowing through Transformer layers.