Mixture of experts (MoE) is a neural network design that activates only part of the model for each input token. The result is a model that can hold hundreds of billions or even trillions of parameters but use a fraction of them per token, cutting compute cost without cutting capacity. For teams moving AI from prototype to production, this matters because cost per token and throughput often decide whether a deployment is viable. As of early 2026, MoE has become the default architecture for frontier models, open-weights and proprietary alike.
What is a mixture of expert architecture?
An MoE architecture uses conditional computation: instead of running every parameter for every input, it routes each token to a small subset of specialized subnetworks. In traditional dense neural networks, every parameter is activated to process every incoming token, meaning that computational costs scale linearly with the model’s total size. An MoE framework alters this dynamic by replacing uniform layers with an algorithmic routing network and a collection of separate, parallel subnetworks called “experts.”
The core idea goes back to the early 1990s with Jacobs et al. (1991) on modular neural networks. The application to modern deep learning came from Shazeer et al. in a 2017 paper, “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer”. Sparsity splits the model’s capacity across specialized subnetworks, keeping the active compute path short while preserving broad knowledge.
How the mixture of experts architecture works
A small router network looks at each incoming token and sends it to one or more expert subnetworks. The routing happens at the token level, so a single sentence can activate a different mix of experts for each word.
The gating network and token routing
The gating network is a trainable layer that scores each expert for a given token and picks the top few. When an input tensor x arrives at an MoE layer containing N experts, the gating network calculates a sparse weight vector G(x). The output y of the layer is a weighted sum of the active experts’ outputs:
Here, Ei (x) is the output of the i-th expert subnetwork. To enforce sparsity, engineers use Top-k gating. In Top-1, the router sets all values in G(x) to zero except the highest, activating exactly one expert. In Top-2, the two highest-scoring experts get non-zero weights and their outputs are blended. Top-2 is the common choice in production models because it preserves stability without doubling compute.
Sparse vs dense layer execution
Modern MoE models are hybrids: attention layers stay dense, only the feed-forward blocks become sparse. The multi-head self-attention mechanisms remain entirely dense so all tokens in a sequence can attend to one another across the full context window.
The MoE swap happens inside the feed-forward network (FFN) blocks that follow the attention layers. Each standard FFN is replaced by a block containing multiple parallel FFNs plus a router. A model with 47 billion total parameters might activate only 13 billion per forward pass, because the router only fires a couple of experts at a time.
Managing expert capacity and load balancing
Without constraints, training tends to collapse onto a few favored experts. This happens when the router sends most tokens to the same subnetwork early in training because it produces lower initial gradients. The router keeps reinforcing that choice, the other experts stay untrained, and the model effectively reduces back to a dense configuration on a much smaller active footprint.
The original fix was an auxiliary load-balancing loss added to the training objective, as described in Google’s GShard architecture documentation. The auxiliary term penalizes uneven token distribution. Newer architectures like DeepSeek-V3 (December 2024) and its successor DeepSeek-V4 (April 2026) drop the auxiliary loss entirely. They add dynamic bias terms to the routing scores instead: an overloaded expert sees its bias drop, an underloaded one sees its bias rise. This balances load without distorting the gradient signal the way auxiliary losses can.
Why mixture of experts matters for enterprise AI deployment
MoE decouples a model’s knowledge capacity from its per-token compute cost. For workloads running at scale (customer support, internal search, document processing), this is the difference between an economically viable deployment and one that bleeds budget on every query.
Reduction in computational overhead
Google’s Generalist Language Model paper put numbers on the savings: a sparse MoE model matched GPT-3 on benchmark performance while using about one-third of the energy for training and roughly half the FLOPs at inference. Lower FLOPs translate to lower electricity bills and less hardware wear in production.
Improvement in inference latency and throughput
Because the router only fires a subset of weights per token, generation speed tracks the active parameter count, not the total. A 47B model with 13B active parameters runs at roughly the speed of a 13B dense model.
Experts can also be distributed across separate GPUs (expert parallelism), letting hardware clusters process different parts of a batch in parallel and pushing total throughput higher.
Scalability of total parameter count
Adding more experts grows the model’s knowledge capacity without changing the active compute path. A dense model has to grow its active layer to store more knowledge, which eventually hits limits on GPU memory bandwidth and interconnect speed. MoE sidesteps that ceiling: total parameters can scale into the hundreds of billions or trillions while per-token compute stays fixed. DeepSeek V4-Pro is the clearest example so far: 1.6 trillion parameters held in memory, 49 billion executed per token, a roughly 32-to-1 ratio between stored and active capacity.
Technical advancements and current model implementations
Since early 2025, nearly all leading frontier models use MoE designs. The releases below trace the shift from experimental architecture to industry standard.
Mixtral 8x7B to Mistral Large 3
Released in December 2023 by Mistral AI, Mixtral 8x7B was the open-weights release that made MoE mainstream: 46.7B total, 12.9B active via Top-2 routing, Apache 2.0. It matched or beat the dense Llama 2 70B on MMLU and GSM8K while running roughly six times faster.
Mistral’s current MoE flagship is Mistral Large 3, released December 2, 2025: 41B active parameters out of 675B total, 256K context window, Apache 2.0. Alongside it, Mistral Small 4 packs 119B total with roughly 6B active per token through MoE, making it one of the cheapest multimodal models in production use. The architectural through-line from Mixtral to Large 3 is the same Mistral pattern of pushing the active-to-total ratio higher while keeping the license permissive.
Llama 4 and Meta’s shift to sparse
Meta moved from dense to MoE with the Llama 4 release in April 2025. Scout activates 17B parameters out of 109B total across 16 experts with a 10M token context window. Maverick activates 17B out of 400B total across 128 experts with a 1M context window. A larger model, Behemoth, was announced but not released. The move matters less for any single benchmark and more as a signal: when the largest open-weights publisher abandons dense architectures, MoE has become the default.
DeepSeek-V4 and advanced routing
DeepSeek-V4, released as a preview on April 24, 2026, is the current flagship and the most aggressive demonstration of MoE scaling to date. It ships in two variants:
- V4-Pro: 1.6 trillion total parameters, 49 billion activated per token
- V4-Flash: 284 billion total, 13 billion activated
Both variants support a 1 million token context window. V4-Pro is currently the largest open-weight model available, ahead of Moonshot’s Kimi K 2.6 (1.1T) and more than double its predecessor DeepSeek-V3 (671B / 37B active).
The DeepSeek architecture introduces several modifications to traditional MoE frameworks, carried forward from V3 and extended in V4:
- Fine-grained expert division. Experts are smaller and more numerous than in earlier MoE models, which lets the router assign specialization with more precision.
- Shared expert isolation. A few experts stay active for every token. They handle universal patterns (basic syntax, common reasoning), so the routed experts don’t each need to carry redundant copies of that baseline knowledge.
- Hybrid attention (new in V4). V4 combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA). At 1M-token context, V4-Pro uses only 27% of the per-token inference FLOPs and 10% of the KV cache compared to V3.2.
- Mixed-precision training (new in V4). V4 uses a mix of FP8 and FP4 precision, with quantization-aware training applied to the MoE expert weights, roughly halving the memory needed to store them versus FP8.
Comparing dense and sparse architectures
Sparse models trade compute for memory. The table below summarizes the practical differences. Note that finding a current frontier dense model for comparison has become harder: Llama 3.3 70B and Mistral Large 2 are recent dense examples, but their successors (Llama 4, Mistral Large 3) both moved to MoE.
| Operational Dimension | Dense Architectures (e.g., Llama 3.3 70B, Mistral Large 2) | Sparse MoE Architectures (e.g., Mistral Large 3, Llama 4 Maverick, DeepSeek-V4) |
|---|---|---|
| Active Parameters per Token | 100% of total parameters are computed for every operation. | Typically 3% to 25% of total parameters are computed (V4-Pro activates roughly 3%, Llama 4 Maverick around 4%). |
| VRAM Requirement for Serving | Scales with total size. | High. The full parameter set must stay loaded in GPU memory, though quantization and CPU offload can mitigate this. |
| Inference Latency | Scales linearly with model size. | Tracks active parameter count, not total. |
| Training & Fine-Tuning Stability | Straightforward, no load balancing needed. | Requires routing safeguards to avoid load imbalance. |
| Knowledge and Task Capacity | Bounded by the size of the active compute path. | Scales by adding experts without raising per-token compute. |
Summary of takeaways
MoE is an engineering trade-off: more memory, less compute per token. The model stays fully loaded in VRAM, but only a slice runs for any given input. For production deployments, that trade lowers per-token serving cost while keeping the broad knowledge needed for complex business tasks.
For teams evaluating whether MoE fits their workload, the practical questions are usually about memory headroom on existing hardware, latency targets, and how much custom domain knowledge needs to be encoded. Want to know if MoE fits your team’s workload? DataNorth can help map those constraints to deployment options: our AI consulting team works through the technical trade offs with you, translates the outcome into a concrete AI strategy, and takes it all the way to development and implementation when you’re ready to build.
Frequently asked questions
Does a mixture of experts model require more GPU memory (VRAM)?
Yes. The router needs instant access to every expert, so the full model has to stay in VRAM even though only a fraction runs per token. Memory footprint tracks total parameters, not active parameters. Quantization and CPU offload can reduce this in practice but don’t close the gap with equivalently-active dense models.
