1.3.3
Decoder-Only vs Encoder-Decoder
What is it?
Transformer models can use different architectural arrangements. Decoder-only models generate text using causal self-attention. Encoder-decoder models first process an input with an encoder and then generate an output with a decoder. Encoder-only models create contextual representations and are often used for understanding tasks.
Key points
1. Encoder-only models process input representations for understanding tasks.
2. Decoder-only models generate sequences using causal attention.
3. Encoder-decoder models combine input processing and output generation.
4. Causal attention prevents a token from using future tokens during generation.
5. Architecture selection depends on the task and model design.
Example: Three Transformer architectures
Text classification:
An encoder-only model can process a review and classify its sentiment.
Text generation:
A decoder-only model can generate a product description from a prompt.
Translation:
An encoder-decoder model can transform a sentence from one language into another.
Real-world application
This distinction helps developers choose an appropriate model architecture for chatbots, classification, translation, summarization and text generation.
Key takeaway
Encoder-only, decoder-only and encoder-decoder models organize Transformer components differently to support different tasks.