1.10.1
Mixture of Experts Architecture
What is it?
A Mixture of Experts model replaces the standard feed-forward network in a Transformer block with multiple expert networks and a router. The router selects a small number of experts for each token, so the model can have a large total parameter count without activating every parameter for every token.
Key points
Each expert is a feed-forward network. The router calculates gating scores for the experts. Top-K routing activates only the K experts with the highest scores. A common configuration activates 2 experts out of 8 to 64. Total model capacity grows with the number of experts, while active computation depends on the selected experts.
Top-2 Expert Routing
Imagine a model with 8 experts. For a particular token, the router assigns scores to all 8 experts. If Expert 3 and Expert 6 receive the highest scores, only those two experts process the token. Their weighted outputs are combined to produce the MoE layer output.
Real-world application
MoE architecture is useful when building or serving large language models that need high parameter capacity while controlling the computation performed for each token.
Key takeaway
MoE increases model capacity by activating only a selected subset of experts for each token.