1.3.7
Positional Encodings
What is it?
Self-attention alone does not inherently identify the order of tokens. Positional information is therefore added to token representations so that the model can use sequence order. Different Transformer models use different positional encoding or positional embedding methods.
Key points
1. Token embeddings represent token identity.
2. Positional information represents sequence position.
3. The original Transformer used sinusoidal positional encodings.
4. Learned positional embeddings are another approach.
5. Rotary Position Embedding (RoPE) is used in many modern models.
6. Positional methods influence how models handle sequence length and position.
Example: Why word order matters
Sentence A:
"The dog chased the cat."
Sentence B:
"The cat chased the dog."
Both sentences contain the same words, but their meanings differ because the word order changes.
Positional information helps a Transformer represent these sequence relationships.
Real-world application
Positional information is important in language generation, long-context processing, document analysis and any task where token order changes meaning.
Key takeaway
Positional information helps Transformers distinguish sequences that contain similar tokens in different orders.