1.3.6
Multi-Head Attention
What is it?
Multi-head attention runs several attention operations in parallel. Each head uses its own learned projections of queries, keys and values. The outputs are combined and projected to create a representation that can capture different relationships within the input.
Key points
1. Multiple attention heads operate in parallel.
2. Each head has separate learned projections.
3. Heads can learn different relationships and patterns.
4. Their outputs are concatenated or combined.
5. A final projection produces the output representation.
6. More heads do not automatically guarantee better model performance.
Example: Multiple attention perspectives
Consider:
"The engineer repaired the machine because it had stopped working."
Different attention heads may learn to focus on different token relationships, such as:
• Relationships between "engineer" and "repaired".
• Relationships between "machine" and "it".
• Relationships between actions and their context.
These are intuitive illustrations, not guaranteed roles for individual heads.
Real-world application
Multi-head attention allows Transformer models to represent several types of token relationships within the same layer.
Key takeaway
Multi-head attention combines multiple parallel attention operations to capture different patterns in a sequence.