1.10.2
Load Balancing
What is it?
Load balancing ensures that tokens are distributed across experts rather than repeatedly routed to only a few of them. Without balancing, some experts may receive too many tokens while others remain underused. This is known as expert collapse.
Key points
Expert collapse reduces the effective use of model capacity. Uneven routing can create GPU workload imbalance. An auxiliary load-balancing loss encourages more uniform expert utilization. The loss considers both the fraction of tokens sent to an expert and its average router probability.
Why Load Balancing Matters
Suppose a model has 8 experts, but the router sends 80% of tokens to only Expert 1 and Expert 2. These experts become overloaded while the remaining experts do little work. A load-balancing objective encourages the router to distribute tokens more evenly.
Real-world application
Load balancing is important in distributed MoE training because experts may be placed on different GPUs. Balanced routing helps avoid overloaded devices and underused compute resources.
Key takeaway
A good MoE router must distribute work effectively across experts, not simply select experts for each token.