← All explorers
Loading explorer…
Read the theory: mixture-of-experts
Read the theory: mixture-of-experts
Route a batch of tokens to experts and watch the router collapse onto a few — starving the rest and dropping tokens — until load-balancing pressure spreads them out.
Each bar is one expert's token load for this batch; the dashed line is the fair share, and the solid line is the capacity cap. With low balance pressure the router collapses onto a few experts (the rest go dead) and over-capacity tokens are dropped. Raise the pressure to spread the load, and adjust the capacity factor to trade memory headroom against dropped tokens.
Illustrative model for building intuition, not a live system.