1.3.5
Self-Attention Mechanism
What is it?
Self-attention allows each token to calculate how much information it should use from other tokens in the same sequence. It uses query, key and value representations. The attention scores determine how strongly each token contributes to the resulting representation.
Key points
1. Each token produces a query, key and value representation.
2. Queries are compared with keys to calculate attention scores.
3. Scores are scaled and normalized using softmax.
4. The resulting weights are applied to value vectors.
5. The weighted values produce contextual representations.
6. Causal masking can restrict access to future tokens.
Example: Understanding a word through context
Consider:
"The bank approved the loan."
The word "bank" appears in a financial context. Attention can help its representation use information from words such as "approved" and "loan".
In another sentence, "They sat beside the river bank", the surrounding words provide a different context.
Real-world application
Self-attention helps LLMs understand relationships across a sentence or sequence, supporting question answering, text generation and document understanding.
Key takeaway
Self-attention lets each token use relevant information from other tokens to build a context-aware representation.