Why does DeepSeek have 671B parameters yet run blazingly fast?
10 min readTraditional large language models use a Dense architecture — every parameter is activated for every token. For example, Llama 70B computes all 70 billion parameters on each forward pass. More parameters means slower inference.
MoE (Mixture of Experts) takes a Sparse approach — the model has a massive total parameter count, but only a small fraction is activated per token. DeepSeek V3 has 671 billion parameters in total, yet only ~37 billion are active per token — just 5.5% of the total.
See how the two architectures differ when processing a single token:
A Dense model is like a general hospital — every department joins the consultation. MoE is like a specialist referral — you only see the relevant experts.
At the heart of MoE is the Router (also called a Gating Network). It is a lightweight neural network that scores each expert for a given token and decides which experts should process it.
The most common strategy is Top-K routing: the router computes a score for every expert and selects the top K (typically K=2). To prevent all tokens from flooding the same expert, a load balancing mechanism ensures even utilization across experts.
Click different tokens to see how the router selects different Top-2 experts for each one:
The router is like a hospital receptionist — it looks at your symptoms and decides whether you should see orthopedics or internal medicine.
In a standard Transformer, each layer has two components: Attention and an FFN (Feed-Forward Network). MoE does not change the Attention layer — it only replaces the FFN.
Specifically, MoE replaces the single FFN with N expert sub-networks (e.g., 64–256 experts). Each expert is structurally identical to the original FFN, but has its own independently learned weights. The Attention layer remains shared — all tokens pass through the same Attention computation.
Standard Transformer vs. MoE Transformer layer structure:
Attention is the all-hands morning meeting. Experts are the specialized work each person does at their own desk afterward.
MoE dramatically reduces compute per token, but there is a catch: all expert weights must reside in memory, even though only 2 are used at a time.
Take DeepSeek V3: its 671B parameters require 400+ GB of memory, yet each token only computes through ~37B parameters' worth of operations. This is the MoE "memory paradox" — low compute cost, high memory footprint. Memory bandwidth becomes the real bottleneck: loading routing tables and selected expert weights from a massive memory pool.
AtomGradient's research on MoE 35B-A3B (35B total, 3B active) illustrates this perfectly: it has the memory footprint of a 35B model but the compute cost of a 3B model.
See how Dense and MoE differ in memory usage vs. compute:
MoE is like a company with 128 offices — only 2 people work at any given time, but you still have to pay rent on the entire building.
MoE has moved from academic concept to production reality. Here are three landmark models:
DeepSeek V3: 671B total / 37B active, 256 experts with Top-8 routing. Competitive with GPT-4 at a fraction of the training cost. MoE enabled DeepSeek to punch far above its compute budget.
Qwen3.5-35B-A3B: 35B total / 3B active, 128 experts with Top-2 routing. Lightweight enough to run on a MacBook — unthinkable just a few years ago.
Mixtral 8x7B: 46.7B total / 12.9B active, 8 experts with Top-2 routing. The pioneer of open-source MoE, proving the architecture's viability in the open ecosystem.
The key insight: MoE enables "train big, run small" — massive knowledge encoded during training, efficient inference at runtime.
| Model | Total Params | Active Params | Experts | Routing |
|---|---|---|---|---|
| DeepSeek V3 | 671B | 37B | 256 | Top-8 |
| Qwen3.5-35B-A3B | 35B | 3B | 128 | Top-2 |
| Mixtral 8x7B | 46.7B | 12.9B | 8 | Top-2 |
Active parameters as % of total:
MoE gives you the knowledge of a team of PhDs, but you only pay an intern's salary.
MoE replaces full-parameter computation with sparse activation, achieving big-model capability at a fraction of the compute cost.
A router dynamically picks the best experts for each token, enabling input-driven division of labor.
MoE memory footprint depends on total parameters, but compute cost depends only on active parameters. Don't confuse the two.
Models like Qwen 35B-A3B prove that MoE can run efficiently on consumer hardware today.
The essence of MoE: not every parameter needs to work for every token — specialization and division of labor.