1.3.11
Attention Pathologies
What is it?
Attention mechanisms are powerful, but their behavior can have limitations. Attention weights do not provide a complete explanation of model reasoning. Some attention heads may be redundant, certain patterns may be poorly represented, and standard attention computation can become expensive for long sequences. Researchers study these limitations to improve reliability and efficiency.
Key points
1. Attention weights are not a complete explanation of model decisions.
2. Some attention heads can learn overlapping patterns.
3. Long sequences increase attention computation and memory requirements.
4. Attention can be sensitive to input patterns and position.
5. Attention visualizations should be interpreted carefully.
6. Different architectures and attention methods address different limitations.
Example: A limitation of attention visualization
Suppose a visualization shows strong attention from one word to another.
It may be tempting to conclude that the first word directly caused the model prediction.
However, the final prediction also depends on other layers, heads, feed-forward transformations and the complete model computation.
An attention map is therefore only one view of model behavior.
Real-world application
Understanding attention limitations is useful when evaluating model reliability, investigating model behavior and designing explainability methods.
Key takeaway
Attention weights can provide useful clues, but they should not be treated as a complete explanation of how an LLM reaches an output.