1.10.4
Notable MoE Models
What is it?
Several well-known language models use Mixture of Experts architectures. They differ in their routing strategy, number of experts, active experts per token and overall model design.
Key points
Mixtral uses sparse expert routing. DeepSeek-V2 uses a Mixture of Experts architecture with specialized routing and shared components. Switch Transformer popularized simplified Top-1 routing. Sparsely-Gated MoE is an important early architecture in modern sparse expert research. Model designs differ, so their parameter counts and active computation should not be compared as if they were identical.
Understanding Sparse Model Capacity
A model may contain a large number of total parameters distributed across many experts, while activating only a small subset for each token. Therefore, total parameters describe model capacity, whereas active parameters provide a better indication of the parameters used for a particular token.
Real-world application
When evaluating MoE models, compare total parameters, active parameters, routing strategy, memory requirements and inference cost rather than relying on total parameter count alone.
Key takeaway
MoE models trade dense computation for sparse expert activation, making routing strategy a key part of their design.