1.3.2
The Original Encoder-Decoder Transformer
What is it?
The original Transformer architecture introduced an encoder-decoder design. The encoder processes the input sequence and creates contextual representations. The decoder generates the output sequence using previously generated tokens and information from the encoder.
Key points
1. The encoder reads and processes the input sequence.
2. The decoder generates the output sequence.
3. Encoder layers contain self-attention and feed-forward networks.
4. Decoder layers use masked self-attention.
5. Cross-attention connects decoder representations to encoder outputs.
6. The architecture was introduced for sequence-to-sequence tasks.
Example: Machine translation
Input:
"Good morning"
The encoder processes the source sentence and creates contextual representations.
The decoder uses these representations to generate a translation such as:
"Buenos días"
The decoder generates the translated sequence step by step.
Real-world application
Encoder-decoder Transformers are useful in translation, summarization and other tasks where an input sequence is transformed into an output sequence.
Key takeaway
The original Transformer uses an encoder to understand the input and a decoder to generate the output.