OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / Mixture of Experts (MoE)
Lesson 60 of 415 6 min read
1.10.1

Mixture of Experts Architecture

What is it?

A Mixture of Experts model replaces the standard feed-forward network in a Transformer block with multiple expert networks and a router. The router selects a small number of experts for each token, so the model can have a large total parameter count without activating every parameter for every token.

Key points

Each expert is a feed-forward network. The router calculates gating scores for the experts. Top-K routing activates only the K experts with the highest scores. A common configuration activates 2 experts out of 8 to 64. Total model capacity grows with the number of experts, while active computation depends on the selected experts.

Top-2 Expert Routing

Imagine a model with 8 experts. For a particular token, the router assigns scores to all 8 experts. If Expert 3 and Expert 6 receive the highest scores, only those two experts process the token. Their weighted outputs are combined to produce the MoE layer output.

Real-world application

MoE architecture is useful when building or serving large language models that need high parameter capacity while controlling the computation performed for each token.
Key takeaway

MoE increases model capacity by activating only a selected subset of experts for each token.