OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / The Transformer Architecture
Lesson 18 of 415 6 min read
1.3.8

Feed-Forward Network (MLP)

What is it?

A feed-forward network, often called an MLP, is a neural network block used within Transformer layers. It applies learned transformations to token representations, usually independently at each sequence position. The block typically expands the hidden dimension, applies a non-linear activation and projects the result back.

Key points

1. The feed-forward network operates on token representations. 2. It commonly contains two linear transformations. 3. A non-linear activation is applied between transformations. 4. The intermediate dimension is often larger than the model hidden dimension. 5. It complements attention by transforming the information at each position.

Example: A simplified feed-forward block

A simplified flow: Input vector ↓ Linear transformation ↓ Activation function ↓ Second linear transformation ↓ Output vector The output is combined with other Transformer operations according to the model architecture.

Real-world application

Feed-forward networks provide the transformation capacity used by LLMs after attention has incorporated contextual information.
Key takeaway

Attention mixes information across tokens, while the feed-forward network transforms each token representation.