OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / Mixture of Experts (MoE)
Lesson 62 of 415 8 min read
1.10.3

Noisy Top-K Gating: Making Discrete Routing Trainable

What is it?

Top-K routing uses a discrete selection operation: the router chooses the K highest-scoring experts. This selection is not directly differentiable, creating a challenge for backpropagation. Noisy Top-K Gating and Gumbel-Softmax are techniques that help make routing trainable.

Key points

In Noisy Top-K Gating, learnable Gaussian noise is added to router logits before expert selection. Noise can give underused experts opportunities to receive tokens and gradient signals. Gumbel-Softmax provides a differentiable relaxation of categorical selection. A straight-through estimator can use hard selection in the forward pass and relaxed gradients in the backward pass.

Noisy Top-K Gating

A router assigns scores to 8 experts. During training, Gaussian noise slightly changes these scores before Top-2 selection. An expert that normally ranks third may occasionally enter the top two. This allows that expert to receive training signals. At inference, the noise can be removed and clean router scores used.

Real-world application

Differentiable routing techniques help train sparse expert models and reduce the risk that some experts never receive useful training signals.
Key takeaway

The challenge is not only choosing experts, but also allowing the router to learn which experts should be selected.