1.3.8
Feed-Forward Network (MLP)
What is it?
A feed-forward network, often called an MLP, is a neural network block used within Transformer layers. It applies learned transformations to token representations, usually independently at each sequence position. The block typically expands the hidden dimension, applies a non-linear activation and projects the result back.
Key points
1. The feed-forward network operates on token representations.
2. It commonly contains two linear transformations.
3. A non-linear activation is applied between transformations.
4. The intermediate dimension is often larger than the model hidden dimension.
5. It complements attention by transforming the information at each position.
Example: A simplified feed-forward block
A simplified flow:
Input vector
↓
Linear transformation
↓
Activation function
↓
Second linear transformation
↓
Output vector
The output is combined with other Transformer operations according to the model architecture.
Real-world application
Feed-forward networks provide the transformation capacity used by LLMs after attention has incorporated contextual information.
Key takeaway
Attention mixes information across tokens, while the feed-forward network transforms each token representation.