TL;DR
Kimi K3 is the largest open model to date — 2.8 trillion parameters — yet it activates only 1.8% of them per token. The interesting story isn’t the benchmarks; it’s that every architectural choice (latent MoE compression, hybrid linear attention, block-grouped residuals) is really an answer to an infrastructure question: how do you move less data between GPUs? And once you understand the architecture, the “expensive” sticker price stops being the right way to measure it. If you run platforms for a living, this model reads less like an ML paper and more like a distributed-systems design doc.
Originally published at portfolio.hagzag.com.
The DeepSeek Moment, Round Two
January 2025 gave us the DeepSeek moment: a Chinese lab matching OpenAI-class benchmarks at a fraction of the GPU rental cost. Eighteen months later, Moonshot AI’s Kimi K3 is doing something similar to Anthropic and OpenAI — but the more interesting shift is happening a layer down, in the economics of serving these models.
Most of the good writing on K3 so far has been the geopolitics take — the open-weight ecosystem closing the cognitive gap, and whether Beijing’s rumored access restrictions are a self-inflicted own-goal (Barbaros Ozturk makes that case well). That framing is right, but it treats the model as a black box. I want to go one layer lower.
I keep coming back to one framing when reading the K3 architecture: this is a model designed by people who priced out the interconnect first. Every headline feature exists to reduce data movement between GPUs. That’s a platform engineering mindset applied to model design, and it’s worth walking through how the pieces fit.
896 Experts, 16 Working at a Time
Mixture of Experts (MoE) became the de-facto industry standard around 2024: instead of pushing every token through the full network, a router activates a small subset of “expert” sub-networks. The efficiency race since then has been about how sparse you can go:
| Model | Expert activation ratio |
|---|---|
| Nemotron 3 Ultra | ~4.3% |
| MiniMax M3 | ~3.1% |
| Inkling (Thinking Machines) | ~3.1% |
| Kimi K3 | ~1.8% |
K3 splits the model into 896 experts and activates just 16 per token. Compare that to its predecessor Kimi K2: they more than doubled the expert count while lowering the activation ratio from 2% to 1.8%. Scale up, activate less — that’s the whole game.
But sparsity on paper doesn’t automatically translate to cheap inference. A model this size, even at reduced precision, spreads its weights across roughly six GPUs in an NVIDIA DGX B300 setup just for the weights — leaving little room for KV cache. Realistic deployments spread the experts across a 64-GPU super-node or an NVL72 rack. Which means every token potentially takes many hops across the interconnect to reach its 16 assigned experts. Each hop is communication overhead.
This is the part most model coverage skips, and it’s exactly where the architecture gets interesting.
Stable LatentMoE: Compress Before You Ship
K3’s answer to the interconnect problem is Stable LatentMoE. Before a token is routed to experts, its embedding is down-projected into a lower-dimensional latent representation. The compressed token travels across GPUs, gets processed by the experts (plus two always-on shared experts), and is projected back up afterward.
If you’ve ever put a compression layer in front of a chatty cross-AZ service call, you already understand the trade: pay a little compute at the edges to save a lot of bandwidth in the middle. Smaller representation → less data on the wire and less compute inside each expert, since the working dimension shrinks too.
The “Stable” half is about training. Picking 16 experts out of 896 is a brutally sensitive selection problem — small router-score variations can destabilize training. Where DeepSeek V3 and Nemotron 3 Ultra nudge the router with bias terms (penalize overused experts, promote underused ones), K3 uses quantile balancing: expert allocation is derived from the distribution of router scores rather than raw values. Think grading on a percentile curve instead of a fixed scale — each expert is selected relative to the field, which keeps the load balanced without fighting the router.
Kimi Delta Attention: The Forbidden Shortcut, Made to Work
The headline operational number — up to 6.3× faster decoding at 1M-token context, without the usual quality penalty — comes from the attention side, via Kimi Delta Attention (KDA). Notably, KDA isn’t new: Moonshot published it in October 2025 and K3 is where it ships at full scale. That release-then-integrate cadence is telling — they de-risked the component nine months before betting the flagship on it.
The shortcut every lab has known about for years is linear attention: keep a fixed-size memory instead of a memory that grows with context. It’s dramatically faster, but a compressed memory recalls precise detail poorly — so historically it traded away quality. The lineage to K3’s fix:
- Linear attention / state-space memory (e.g., Mamba-2 in Nemotron 3): replace expensive attention layers with a recurrent memory cell — new information writes in, old memory decays at a learned rate. Effectively an RNN-style memory.
- Gated DeltaNet: instead of rewriting memory wholesale, compute the error between prediction and new value and apply only the correction. More efficient updates — but the paper itself admits hardware-efficient implementation remained an open challenge.
- KDA: gives per-channel control over retention and forgetting. When new information contradicts something already stored, KDA rewrites the old entry rather than piling more on top — and it can decay memory channel by channel at different rates, which is exactly what Gated DeltaNet couldn’t do.
The quality problem gets solved by not going all-in on linear attention. K3 interleaves a 3:1 hybrid — three fast KDA layers per one full global-attention layer (a gated MLA) with perfect recall, a ratio picked via ablation. Fast memory for the bulk of the work, exact memory where it counts. And because RNN-style memory historically trains poorly (sequential dependence kills parallelism), they use a chunk-wise parallel algorithm to keep training hardware-efficient.
Attention Residuals: The Assembly Line Problem
The web of red lines in the architecture diagram is attention residual — released in March 2026, integrated four months later. The clearest way to see why it matters: think of the model as an assembly line where every token passes through hundreds of stations (layers), each adding a bit of work.
In a standard transformer, every station dumps its output into one shared scratchpad — the residual stream. By the time you reach layer 100 or layer 1,000, any single early insight is a drop in the ocean; the model can barely hear its own good work from 90 layers ago. Attention residuals fix this by keeping each layer’s contribution separate and letting later layers use attention to selectively pull the earlier results that matter. An insight from layer 5 can arrive at layer 100 fully intact, grabbed directly rather than fished out of the mix.
The reported payoff is striking: the model trains as if it were given roughly 25% more compute, for under 2% additional latency. In a compute-constrained era, that is an enormous lever. K3’s twist is grouping layers into logical blocks so these cross-layer connections don’t explode in cost — more expressiveness, without turning the residual stream into yet another interconnect burden. Same design tension as everything else in this model: information flow versus bandwidth.
The Metric That Actually Bills You: Cost per Task
Here’s where the architecture pays off in a way the pricing coverage misses. Everyone quotes K3’s sticker price — $3 per million input tokens, $15 per million output — and concludes it’s a premium model to use sparingly. But per-token pricing is the wrong unit. What bills you is cost per task: the total spend to actually finish a job, which depends on how many turns, retries, and tokens the model burns getting there.
A cheaper-per-token model that needs three times as many turns to close a task is not cheaper. On agentic benchmarks like DeepSWE and BrowseComp, this inverts the leaderboard: models that look mid-priced per token can end up the most expensive per completed task, while K3 reportedly lands one of the highest scores at one of the lowest total dollar costs. (These figures come from Moonshot-aligned reporting, so treat them as directional until independent evals land — but the framing is correct regardless of whose numbers you trust.)
There’s a practitioner corollary here worth stealing: routing and planner/worker splits. Reported experiments rebuilding a real codebase found that using a big, slow model as the planner and a small, fast model as the worker hit the same result for roughly 9–10× less than running the big model end-to-end. If you’re deploying K3, the architecture rewards treating it as a coordinator, not a workhorse.
What the “Obvious” Reading Gets Wrong
The obvious take is “Chinese lab catches up, and the fight is now about open weights and export policy.” True as far as it goes — but K3’s innovations (latent compression before routing, quantile-based load balancing, channel-granular memory decay, block-grouped residuals) are all systems optimizations. The frontier fight is quietly moving from “who has the most compute” to “who wastes the least of it,” and that’s an architecture story the geopolitics coverage skips.
It also reframes the pricing. The sticker price is set by the physics of serving — 2.8T of weights sharded across a 64-GPU super-node, KV-cache fighting the weights for HBM, every token hopping the interconnect to reach its 16 experts. That’s today’s topology on today’s hardware. Once the weights are public, 20-plus inference providers can serve them — some fast and pricey, some slow and cheap — and the cost-per-task math shifts under everyone. That optionality is something no closed model can offer.
Which sets up the real irony. If the compression and hybrid-attention wins hold up under independent serving, a Chinese model architected for compute scarcity may end up running cheapest on the exact US hardware it was designed to live without. Talk of banning it would forfeit precisely that advantage.
Conclusion
If you strip away the AI framing, Kimi K3 is a case study in designing for the constraint that actually bills you: data movement. Sparse activation, latent compression, hybrid attention, grouped residuals — each is a bandwidth decision wearing an ML costume. And the right way to price it isn’t per token; it’s per task.
The geopolitics writers are right that the ultimate victor is whoever lets the world build on their foundation. But a foundation’s economics aren’t set by who’s allowed to use it — they’re set by the interconnect underneath it, and by who serves it most efficiently. Whoever runs this architecture on the fastest fabric, at the lowest cost per task, owns the leverage. That’s not a policy question. It’s a platform question — the same lesson infrastructure learned a decade ago: the network is the computer, and the interconnect is the invoice.
Discussion