Inside Kimi K3's Two Architectural Pillars: KDA and AttnRes, Explained From Zero

中文版 / Chinese Version: 拆解 Kimi K3 的两大架构支柱：KDA 与 AttnRes，讲到小白也能懂

Beginner-friendly · Analogies all the way down · By the end you’ll know how 2.8T stays standing

In the Kimi K3 release deep-dive I left myself a homework assignment: a proper architectural explainer of KDA and AttnRes. Good news — no need to wait for the official technical report. Both papers have been public for a while: KDA is arXiv:2510.26692, AttnRes is arXiv:2603.15031. This article assumes zero deep-learning background. I promise to explain every step with an analogy. Read on with confidence.

Prerequisites: How a Large Model “Reads”

Spend three minutes building two mental pictures. Everything that follows depends on them.

Picture 1: Attention = an Open-Book Exam

A large model generates text one word at a time. Before emitting each new word, it does one thing: it looks back over every word produced so far and judges which ones matter most right now. That is the famous attention mechanism.

Think of it as an open-book exam: for every question (every word generated), you may spread the entire book (everything before) on the desk and flip through it. When the model reads “Xiao Ming gave the apple to Xiao Hong, and she was happy,” understanding who “she” is requires flipping back to “Xiao Hong.” Flip accurately, understand deeply — this is the root of the Transformer’s power.

Picture 2: Residual Connections = “CC the Original” in a Game of Telephone

Large models are stacked layer upon layer — a K3-class model has dozens to hundreds. Information flowing from layer 1 to the last layer is a game of telephone: each layer processes the message and passes it on. You’ve played telephone — by the tenth person, the message is mangled.

Deep learning’s fix is the residual connection: each layer doesn’t replace the content — it adds its processing on top of the original and passes both down. It’s like attaching the original recording to every retelling: the message can’t be lost no matter how many hands it passes through. This design has been the lifeline of deep networks since ResNet in 2015.

💡 These two pictures are all you need: attention = flipping the book in an open-book exam, governing how the model finds information within a sequence; residual connections = passing the original down, governing how information flows between layers. KDA rebuilds the former; AttnRes rebuilds the latter.

Two Growing Pains in This Classic Design

Pain 1: The Thicker the Book, the Less You Can Afford to Flip It

The open-book exam has a fatal flaw: every question requires flipping through the whole book. Fine at 100 pages. At 1,000,000 pages (a 1M-token context — roughly dozens of novels), flipping the entire book for every single word is astronomical. Mathematically this is quadratic complexity: 10x longer text, 100x more compute.

Worse, to flip fast the model must keep the whole book at hand — every historical token keeps an “index card” in GPU memory (the KV cache). Push context to the millions and those cards alone can blow out a server’s worth of VRAM. So the first wall in long context isn’t “can the model understand” — it’s “can you afford the power bill and the VRAM.”

Pain 2: The Deeper the Stack, the Quieter Each New Layer

Now the residual side. “Every layer’s output gets added to the trunk” sounds lovely, until you do the arithmetic: layer 1 adds its bit, layer 2 adds its bit… by layer 80 the trunk carries 80 layers of accumulated output, each with a fixed weight of exactly 1 — nobody more important than anybody else.

The AttnRes paper states the consequence bluntly: trunk values grow uncontrollably with depth while each new layer’s share of the total shrinks. Analogy: adding salt to an ever-thickening soup — the bigger the pot, the less one spoonful registers. Layer 80’s hard-won output lands in 79 layers of accumulated stock and barely makes a splash. This directly caps the returns of going deeper — you stack the layers, and the back half phones it in.

💡 The diagnosis in one line: horizontally (longer text), the flipping bill is quadratic; vertically (deeper network), new layers get diluted by the backlog. KDA treats the horizontal disease; AttnRes treats the vertical one. Let’s take them one at a time.

KDA: From “Flipping the Whole Book” to “Keeping One Page of Smart Notes”

Step 1: Understand the Linear-Attention Lineage

If flipping the whole book is too expensive, why not change the exam format — closed book, but you may bring one A4 cheat sheet? That’s linear attention: don’t store the full history; compress the essentials into a fixed-size memory board as you read, updating as you go.

The benefits are immediate: no matter how thick the book, the cheat sheet stays one page — compute grows linearly with length (10x longer costs 10x, not 100x), memory footprint stays constant, and the KV-cache-blowout disaster simply vanishes.

The cost is equally obvious: the page is only so big — what happens when it’s full? Writing new content means overwriting old content. “What to remember, what to forget, how to erase” has been this lineage’s central struggle for over a decade. In fact, past linear-attention models consistently lost to full attention precisely here — their notes were too sloppy.

Step 2: Three Generations of Smarter Note-Taking

• Generation 1 (early linear attention): pile it on. New information gets stacked straight onto the board; old and new smear together, getting blurrier with every update.
• Generation 2 (the Delta rule): erase before writing. Before writing a new note, find the related old content on the sheet, erase the outdated part, then write. If the sheet says “meeting on Tuesday” and news arrives that it moved to Thursday, the Delta rule strikes out “Tuesday” before writing “Thursday” — instead of keeping both entries to fight it out. The “Delta” in KDA comes from here.
• Generation 3 (Gated DeltaNet): add a forgetting valve. Install a gate on the memory board so old information actively fades over time, making room for the new. But the valve is crude — one switch for the entire board: when things fade, everything fades together.

Step 3: KDA’s Key Move — Replace One Valve With a Row of Valves

KDA (Kimi Delta Attention) stands on Gated DeltaNet’s shoulders, and its key step is making the gating fine-grained: the memory board is split into many channels, each with its own independent forgetting rate.

Why does that matter? Imagine the model reading a large codebase. Some information is long-term skeleton (the project’s directory structure, core interfaces) that should persist; some is read-and-burn detail (temporary variables inside the current function) that should clear out when done. With one valve for everything, long-term and short-term memory could only fade together. KDA lets each fade at its own pace — forget the important slowly, forget the trivial fast. In the paper’s words: the limited memory state gets “used more efficiently.” One A4 cheat sheet can now hold the essence of an entire project.

Left: open-book exam — every new word reviews the full history. Right: one cheat sheet + KDA per-channel forgetting | Original illustration

Step 4: Elegant Math Is Useless If the GPU Can’t Run It

Here’s a point outsiders often miss: GPUs love “large, regular matrix multiplications” and hate “processing one token at a time in sequence.” Linear attention is inherently sequential (like an RNN) — a naive implementation starves the GPU.

KDA’s answer is a chunkwise algorithm: cut the sequence into small blocks, reorganize the computation within each block into the big matrix multiplications GPUs love, and only pass state sequentially between blocks. Technically it uses a specialized variant of DPLR (diagonal-plus-low-rank) transition matrices — you don’t need to know the term, only the conclusion: it cuts a large slice of compute versus the general form while preserving the classic “erase-then-write” behavior. Between mathematical elegance and hardware efficiency, Kimi didn’t hesitate to pick the latter — a pragmatism that runs through its entire stack.

Step 5: Not Purebred — Hybrid

“However smart the cheat sheet, sometimes you need to flip the actual book” — say, retrieving one exact number from a million tokens. So Kimi didn’t bet everything on linear attention. It runs a layered hybrid at a 3:1 ratio: every 3 KDA layers (cheat sheet — fast, cheap) are paired with 1 full-attention layer (flip the book — precise). Like an exam where most questions run on notes and brainpower, and only the few big ones justify opening the book.

The validation model is Kimi Linear: 48B total parameters, 3B active. The team raced it against a pure full-attention model using identical data, recipe, and training budget.

Paper Figure 1. Left: Kimi Linear (blue dot, top right) scores highest on 128k long-context while decoding nearly 4x faster. Right: the longer the context, the bigger the edge — at 1M tokens, per-token latency is 1/6.3 of standard attention (MLA, dashed line) | Source: arXiv:2510.26692

✦ Results (Kimi Linear paper): One — wins across all three settings: short context, long context, and RL. The paper claims this is the first time a linear-attention architecture has beaten standard attention across the board in a fair comparison. Two — KV-cache memory reduced by up to 75%. Three — generation at 1M-token context up to 6x faster (the K3 blog says 6.3x).

Underline the phrase “fair comparison” — it’s the most valuable part. Linear-attention papers used to cherry-pick favorable arenas. This time it was same track, same rules, head-on — and it still won. That’s why K3 dares to put KDA inside a 2.8-trillion-parameter flagship: it’s not a cost-saving downgrade — it’s faster and better.

AttnRes: From “Running Ledger” to “Notebook With an Index”

First, Sharpen the Problem

Back to the soup. The standard residual rule: every layer’s output is poured into the trunk, unconditionally, at equal weight. When layer 80 takes over, the pot holds a stew of the previous 79 layers — it cannot say “I only want the basic syntax features layer 3 extracted, hold the middle layers.” Want layer 3’s output? Sorry — you drink the whole soup.

Two concrete harms follow. First, trunk values snowball with depth (everyone adds, nobody subtracts), and large values make training unstable. Second, deeper layers contribute a shrinking share — in a hundred-layer model, the back half may be barely whispering. You paid to train a hundred layers; maybe sixty do real work. That’s compute burned for nothing.

AttnRes’s Answer: Let Every Layer Order à la Carte

The idea behind AttnRes (Attention Residuals) is disarmingly simple: if attention lets the model pick what matters within a sentence, why not use the same mechanism to let each layer pick what matters among all previous layers’ outputs?

Three steps — the standard attention toolkit, rotated from horizontal to vertical:

• Score: layer 80, holding its current needs, gives each of layers 1–79’s outputs a relevance score;
• Normalize: softmax turns the scores into weights summing to 1 (a budget allocation: layer 3 gets 40%, layer 77 gets 25%, the rest split the remainder);
• Weighted sum: mix the layers’ outputs by those weights and use the result as input.

The crucial part: the weights aren’t hard-coded — they’re learned, and they shift dynamically with the input. Processing code might weight the syntax layers; doing math might weight the reasoning layers. In one analogy: the residual stream stops being a “ledger you can only append to and never flip back through” and becomes a notebook with an index — when layer 80 wants layer 3’s original notes, it looks them up and retrieves the original, no fishing in the soup.

Left: standard residual — everything poured in at equal weight. Right: AttnRes — score and order by content | Original illustration

Paper Figure 1: (a) standard residual, (b) full AttnRes — the extra red back-connections at each layer are the “flip back to any earlier layer” channels, (c) the chunked Block AttnRes | Source: arXiv:2603.15031

The effect? The paper reports the dilution problem clearly relieved: per-layer output magnitudes and gradients become uniform along the depth axis — in plain language, in a hundred-layer model, all hundred layers do real work; nobody coasts. And the gain reproduces consistently across model sizes (the scaling-law experiments), so it’s not a small-model fluke.

The Cost? You’ve Probably Already Guessed

“Every layer looks back at all previous layers” — then all previous layers’ outputs must stay resident in memory, an enormous cost as depth grows. Worse, large-model training pipelines layers across machines (pipeline parallelism), and cross-layer lookback means machines constantly shipping data to each other. Academia proposed “every layer sees every layer” ideas long ago; they never made it into large models precisely because of these two bills.

Kimi balanced the books with three moves, each easy to grasp:

• Block AttnRes (chunking): don’t let every layer see every layer — group layers (say, 8 per group) and attend only over group-level summaries. Like research where you skim chapter abstracts instead of every page: memory drops sharply, most of the benefit stays;
• Cached pipeline communication: pre-cache the data that must cross machines so compute never stalls waiting on transfers;
• Two-phase computation: split the attention computation into two staggered steps to further hide the overhead.

✦ The final bill: extra compute — under 2% (per the K3 blog). Training-efficiency gain — about 25%: the paper’s phrasing is that training with Block AttnRes reaches a loss the baseline would need 1.25x the compute to match. Pay $2, collect $25 — that’s why it made it into K3. The official implementation is open source at github.com/MoonshotAI/Attention-Residuals, positioned as a drop-in replacement for standard residual connections.

Zooming Out: One Idea, Two Directions

Put the two components side by side and a beautiful symmetry appears:

	KDA	AttnRes
Direction	Horizontal (sequence length)	Vertical (network depth)
Disease treated	Quadratic flipping bill	Deep-layer dilution
Analogy	Smart cheat sheet: per-channel memory & forgetting	Indexed notebook: retrieve any layer on demand
Payoff	−75% memory, 6.3x faster 1M decoding	+25% training efficiency, <2% overhead

See the common thread? Both do the same thing: replace “fixed, undiscriminating” information processing with “learned, content-dependent” selection. KDA lets the model choose what to remember and forget; AttnRes lets it choose which layers to use. Attention rose to power in the first place by “letting the model choose” its way past a generation of fixed-rule architectures — now the same idea has been generalized to memory management and residual connections.

Back in K3’s context, these two components are the precondition for 2.8T + 1M context being viable at all: KDA makes million-token inference economically feasible — the reason Moonshot can price 1M context at Claude Sonnet 5 rates and still profit is largely KDA plus prefix caching; AttnRes ensures every layer earns its keep as the model grows deeper and larger. When Moonshot says K3’s scaling efficiency is 2.5x that of K2 — these two are the main contributors.

Bonus: The Community Relay Has Already Started

As I wrote last time, the value of an open architecture is letting the whole world reproduce, remix, and falsify it. Four months after the AttnRes paper, arXiv already shows two kinds of follow-up:

• Fault-finders: one study (arXiv:2605.17887) finds AttnRes aggravates outlier activations, complicating low-precision quantized deployment — a practical warning for teams that compress models;
• Extenders: Delta Attention Residuals (arXiv:2605.18855) argues standard AttnRes retrieves too much redundancy and proposes a fix — essentially applying KDA’s “erase-then-write” idea along the depth axis.

One critique, one extension — that’s what a healthy open-research ecosystem looks like. With a closed lab’s architectural innovations, the outside world doesn’t even know whether they’re used, let alone help find their bugs.

Closing: K3 Was Not “Stacked” Into Existence

Once you understand these two components, the number “2.8 trillion parameters” reads differently. It’s not a product of brute wealth: Kimi first validated KDA’s and AttnRes’s gains rigorously on a 48-billion-parameter model, wrote the GPU kernels, finished the inference-framework integration — and only then dared to scale the architecture up by dozens of times to 2.8T.

Prove the architecture on a small model first; then use the architecture to unlock scale. Under compute constraints, that may be a more instructive path than “brute force makes miracles.” When the weights and the full technical report land on July 27, there should be fresh material on how KDA and AttnRes behave at 2.8T scale. We’ll pick it up then.

---

This article uses heavy analogy for accessibility; precise definitions live in the original papers. Full ablation data at K3 (2.8T) scale awaits the forthcoming technical report.