Modern Transformers

Few people have firsthand experience with the pretraining of a frontier-scale language model. For most people working with language models, the original Transformer (Vaswani et al., 2017) and the basic decoder-only architecture used by the first GPT model (Radford et al., 2018) are already very familiar. However, for those without access to large-scale training, keeping up with new model releases does not necessarily mean examining each change to their internal architecture. Leading open models still use the Transformer as their foundation; this post focuses on decoder-only models. Many of their key components have evolved substantially. This post tries to collect and organize the architectural designs of several leading open models, explaining what has changed in modern Transformers, how the components are now combined, and the intuition behind these designs.

A Quick Recap: The Decoder-Only Transformer

Before looking at the changes, it helps to trace the familiar decoder-only Transformer once from input to output. The diagram below uses the first GPT model as a compact historical baseline and follows one continuous path through the entire model.

Start at the top. Each token ID indexes a token embedding, while its position indexes a position embedding. These two vectors have the same width and are added to form $h_0$, a sequence of $T$ hidden vectors. GPT-1 uses learned absolute position embeddings here, whereas the original Transformer used fixed sinusoidal positional encodings. The resulting $h_0$ is the input to the first decoder block. Every later block receives the previous block’s output, so block $l$ maps $h_{l-1}$ to $h_l$.

Within a GPT-1 block, causal self-attention lets the hidden state at position $t$ gather information from positions up to $t$. Its output is added to the block input and then normalized. The same MLP, with parameters shared across all sequence positions, is then applied independently to each position. A second residual addition and LayerNorm produce $h_l$. This is the historical post-norm ordering: $n_l = \operatorname{LayerNorm}(h_{l-1} + \operatorname{Attention}(h_{l-1}))$, followed by $h_l = \operatorname{LayerNorm}(n_l + \operatorname{MLP}(n_l))$.

After $L$ blocks, the model has final hidden states $h_L \in \mathbb{R}^{T \times d}$. GPT-1 reuses the token embedding matrix as its output projection: multiplying every hidden state by $W_e^\top$ produces a logits matrix in $\mathbb{R}^{T \times \lvert V \rvert}$. Row $t$ contains one score for every vocabulary token, treating each as a candidate for the token after position $t$.

The same logits serve two different uses. During training, row 1 is compared with $x_2$, row 2 with $x_3$, and so on through row $T-1$ with $x_T$. Equivalently, the prediction after $x_t$ is trained against the next token $x_{t+1}$. During autoregressive decoding, only the newest logits row is needed. Softmax converts that row into a next-token distribution, and the decoding rule selects the next token, for example by greedy choice or sampling.

This familiar path provides the reference point for the rest of the post. Modern models preserve it while changing how position enters, how attention routes and stores information, how feed-forward capacity is allocated, and how residual blocks are organized.

The Models at a Glance

With the historical baseline in view, we can now ask what a modern decoder-only Transformer actually looks like. To make that question concrete, I selected six of today’s most capable open models: Kimi K2.5 (Moonshot AI, 2026), Qwen3.5 (Qwen Team, 2026), DeepSeek V4-Pro (DeepSeek-AI, 2026), MAI-Thinking-1 (Microsoft AI Team, 2026), GLM-5.2 (Z.ai, 2026), and Inkling (Thinking Machines Lab, 2026). The table below collects the main architectural techniques used by each model.

The table groups those changes into five areas. Position covers rotary position embeddings (RoPE), partial RoPE, YaRN, and relative position bias. Attention covers grouped-query attention (GQA), multi-head latent attention (MLA), Gated DeltaNet, Compressed Sparse Attention (CSA), Heavily Compressed Attention (HCA), and DeepSeek Sparse Attention (DSA). The MLP column collects SwiGLU and mixture-of-experts (MoE) designs, while normalization includes root mean square layer normalization (RMSNorm) and query-key normalization (QK-Norm). The final column holds techniques that cross component boundaries, including query-key clipping (QK-Clip), multi-token prediction (MTP), manifold-constrained Hyper-Connections (mHC), attention sinks, KVShare, and short convolutions.

The rest of this post works through these categories one at a time, explaining what each change does and why it can work.

Positional Information

A token embedding identifies a token, not an occurrence of that token. Before any contextual layer, the word “the” receives the same vector wherever it appears. The query, key, and value projections are also shared across sequence positions, so applying them does not introduce a row index. For a query at position $t$, let $\mathcal V(t)$ be the set of source positions that the attention mask allows it to see, and let $d_h$ be the width of one attention head. Attention aggregates those visible key-value pairs as

\[\operatorname{Attention}_t =\sum_{s\in\mathcal V(t)} \frac{\exp(q_t^\top k_s/\sqrt{d_h})} {\sum_{j\in\mathcal V(t)}\exp(q_t^\top k_j/\sqrt{d_h})}v_s.\]

If two visible pairs $(k_s,v_s)$ exchange places, the terms in this sum exchange places and the result stays unchanged. Attention can distinguish them only when their vectors carry some clue about where they came from. A decoder’s causal mask does provide one piece of order: $\mathcal V(t)$ contains only positions up to $t$. Within that visible prefix, the mask does not label one key as “one token ago” and another as “one thousand tokens ago.” In the first layer, two occurrences of the same token therefore produce the same key and value regardless of their distance from the query.

Language needs more information than membership in a prefix. “Dog bites man” and “man bites dog” contain the same tokens and express different events. Nearby tokens often form local syntax, while a name and its later pronoun may be separated by a long span. A positional mechanism gives the model usable coordinates for these relationships. The main architectural question is where those coordinates enter: they can be added to the input, folded into the query-key geometry, or added directly to the attention score.

The original Transformer and GPT used the first option. Their position vectors entered once, before the first Transformer block.

Position Embeddings

Both early designs assign every index $t$ a vector $p_t$ with the same width $d$ as the token embedding and form

\[h_t=e(x_t)+p_t.\]

Why is addition enough? Write the token embedding $e(x_t)$ more compactly as $e_t$. Using column vectors, the query at position $t$ and key at position $s$ are

\[q_t=W_Qh_t=W_Q(e_t+p_t), \qquad k_s=W_Kh_s=W_K(e_s+p_s).\]

Let $A=W_Q^\top W_K$ abbreviate the two learned projections inside their dot product. Expanding the attention score gives

\[\begin{aligned} q_t^\top k_s &=(e_t+p_t)^\top A(e_s+p_s) \\\\ &=e_t^\top A e_s +e_t^\top A p_s +p_t^\top A e_s +p_t^\top A p_s. \end{aligned}\]

The first term compares token content. The middle terms let token content interact with an absolute location. The final term lets attention compare two locations even when their token embeddings are identical. Addition works because every subsequent learned projection mixes the token and position components. The model can learn that a word behaves differently near the beginning of a document, that certain tokens often follow a nearby delimiter, or that two absolute locations should interact in a particular way.

Learned absolute embeddings. The simplest approach gives every slot in the context its own trainable vector. The token embedding answers “which token is this?”, while the position embedding answers “which slot does this occurrence occupy?” If the same token $x$ appears at positions 7 and 23, the two initial hidden states are

\[h_7=e(x)+p_7, \qquad h_{23}=e(x)+p_{23}.\]

The token component is identical, while $p_7$ and $p_{23}$ distinguish the two occurrences. The vectors are called absolute because each one belongs to a specific index, and learned because gradient descent updates them with the rest of the model. GPT-1 trained on 512-token sequences with hidden width 768, so it learned 512 position vectors of 768 numbers each. Implementations store them together as a lookup table $P\in\mathbb R^{512\times768}$ and select row $P_t$ at position $t$.

This flexibility also explains the limitation. $p_{100}$ and $p_{101}$ begin as unrelated parameters, so the model must learn from data that they represent neighboring locations. Position 512 has no corresponding trained vector. Learned absolute embeddings provide flexible coordinates inside the trained context, while relative displacement and longer unseen positions receive no built-in structure.

Sinusoidal embeddings. The original Transformer replaced the learned lookup table with a deterministic construction. It still creates one $d$-dimensional vector for every position, but the coordinates now follow smooth, repeating curves instead of being trained independently. Assuming $d$ is even, the construction uses $d/2$ frequencies. For frequency index $i=0,\ldots,d/2-1$, one cosine coordinate and one sine coordinate form a pair:

\[p_t^{(i)} = \begin{bmatrix} \cos(\omega_i t) \\\\ \sin(\omega_i t) \end{bmatrix}, \qquad \omega_i=10000^{-2i/d}.\]

The frequency $\omega_i$ tells us how many radians the pair rotates when the position advances by one token. Equivalently, its wavelength $\lambda_i=2\pi/\omega_i$ is the number of token steps required for one full turn. For a toy width $d=6$, three frequency pairs are placed next to one another:

\[p_t= \begin{bmatrix} \cos(\omega_0t),& \sin(\omega_0t),& \cos(\omega_1t),& \sin(\omega_1t),& \cos(\omega_2t),& \sin(\omega_2t) \end{bmatrix}^{\top}.\]

This entire $d$-coordinate vector is $p_t$, so it is added to the token embedding as $h_t=e(x_t)+p_t$. The original base Transformer used width 512, which means that every position received a vector assembled from 256 such pairs. The pairs use different frequencies, yet they all enter the model through the same single addition.

Why use sine and cosine for each pair? A shift in position becomes a rotation that is independent of the starting index. For any displacement $\Delta$,

\[p_{t+\Delta}^{(i)} = \begin{bmatrix} \cos(\omega_i\Delta)&-\sin(\omega_i\Delta) \\\\ \sin(\omega_i\Delta)&\cos(\omega_i\Delta) \end{bmatrix} p_t^{(i)} =R_i(\Delta)p_t^{(i)}.\]

The same matrix $R_i(\Delta)$ moves the pair forward by $\Delta$ at every position. Relative displacement also appears when two pairs are compared:

\[\begin{aligned} \left(p_t^{(i)}\right)^\top p_s^{(i)} &=\cos(\omega_i t)\cos(\omega_i s) +\sin(\omega_i t)\sin(\omega_i s) \\\\ &=\cos(\omega_i t-\omega_i s) \\\\ &=\cos\!\left(\omega_i(t-s)\right). \end{aligned}\]

The cosine subtraction identity produces the second line, so the final comparison depends on $t-s$. If both positions move forward by the same amount, the comparison stays unchanged. Cosine alone is symmetric in the sign of the displacement, but the two coordinates also provide the cross terms

\[\sin(\omega_i t)\cos(\omega_i s) -\cos(\omega_i t)\sin(\omega_i s) =\sin\!\left(\omega_i(t-s)\right).\]

This quantity changes sign when the order of the two positions is reversed. Learned query and key projections can form combinations of these same coordinate products, so a sine-cosine pair contains information about both the size and direction of a relative displacement.

One sine-cosine pair eventually travels around its circle and repeats. Its frequency controls two linked quantities: how much its phase changes per token and how many tokens it takes to complete a turn. A complete positional encoding combines pairs across a range of frequencies. Figure 2 shows that spectrum for $d=16$ and 512 positions, using the original frequency base of 10,000. Each row traces one coordinate as $t$ changes, while each column contains the complete vector $p_t$.

The narrow bands near the top come from short wavelengths; the broad bands near the bottom come from long ones. Moving from $t$ to $t+\Delta$ advances pair $i$ by the angle $\omega_i\Delta$, and its Euclidean change is

\[\left\lVert p_{t+\Delta}^{(i)}-p_t^{(i)}\right\rVert =2\left\lvert\sin\!\left(\frac{\omega_i\Delta}{2}\right)\right\rvert \approx \omega_i\lvert\Delta\rvert \qquad \text{when }\lvert\omega_i\Delta\rvert\text{ is small}.\]

For the same small $\Delta$, a larger frequency produces a larger change. The fastest pair in the figure uses $\omega=1$, so one token rotates it by one radian, about $57^\circ$, and a full turn takes about $6.3$ token steps. The slowest pair uses $\omega\approx0.000316$: one token rotates it by only about $0.018^\circ$, and a full turn takes about 20,000 tokens. The upper pairs make nearby offsets produce visibly different phases, although they cycle quickly. The lower pairs barely move between neighboring positions, so they cannot separate those positions on their own; they provide a slowly changing reference across much longer spans.

The combination resembles reading fast and slow hands on a clock. A fast hand resolves small steps within a short cycle; a slow hand places that cycle within a broader one. Likewise, the full position vector combines local sensitivity with longer-range phase, and two positions coincide only when every pair lines up. The base Transformer uses 256 pairs whose wavelengths span roughly 6 to 60,000 token steps. For a displacement $\Delta$, the values $\cos(\omega_i\Delta)$ and $\sin(\omega_i\Delta)$ across all frequencies form a multi-scale relative signature that learned projections can combine. Larger token distances do not have to produce larger distances between position vectors; the code is periodic rather than monotonic.

Sinusoidal embedding is still an absolute encoding because $p_t$ is generated from the absolute index and added at the input. The full attention score continues to contain the separate content-position terms shown earlier. Its useful guarantee is narrower: shifts have a shared geometric structure, and relative displacement can be recovered from that structure. The constant 10,000 controls the range of wavelengths; it is a design choice rather than a mathematically unique value.

The sinusoidal construction gives position a useful geometry, although that geometry enters only once, when the input embedding is formed. RoPE carries the same idea into the attention calculation itself, where relative position can directly change which key a query chooses to read.

Position in Modern Transformers

RoPE. Rotary Position Embedding moves the rotational structure into the query-key comparison itself (Su et al., 2021). The frequency schedule from Figure 2 reappears here. In the sinusoidal embedding, a column of sine-cosine values forms $p_t$ and is added to the token embedding. In RoPE, those values parameterize rotations applied to coordinate pairs in the query and key.

RoPE starts from a simple target. If $f(q_t,t)$ applies position $t$ to a query and $f(k_s,s)$ does the same for a key, their dot product should carry the content comparison while position enters through the displacement $s-t$:

\[f(q_t,t)^\top f(k_s,s) =g(q_t,k_s,s-t).\]

The requirement keeps both content vectors and reduces the two position indices to their relative displacement. Rotations have exactly this composition rule. RoPE groups the query and key coordinates into pairs and rotates each pair by an angle determined by its position. For one frequency, the rotation at position $t$ is

\[R_t^{(i)} = \begin{bmatrix} \cos(\omega_i t) & -\sin(\omega_i t) \\\\ \sin(\omega_i t) & \cos(\omega_i t) \end{bmatrix}.\]

Each coordinate pair uses its own frequency, just as it does in a sinusoidal embedding. Placing these two-dimensional rotations along the diagonal of one larger matrix lets every pair rotate independently; call that matrix $R_t$. RoPE computes

\[\widetilde q_t = R_t q_t, \qquad \widetilde k_s = R_s k_s.\]

At frequency $i$, the query at position $t$ has turned by $\omega_i t$ and the key at position $s$ has turned by $\omega_i s$. Attention then compares the complete rotated vectors with a dot product. Transposing a rotation reverses its angle, so $R_t^\top=R_{-t}$. Applying one rotation after another adds their angles, so $R_aR_b=R_{a+b}$. The two absolute rotations therefore combine as

\[\widetilde q_t^{\top}\widetilde k_s = q_t^{\top}R_t^{\top}R_s k_s = q_t^{\top}R_{s-t}k_s.\]

The final rotation depends only on $s-t$. This is RoPE’s relative-position property: a query at position 5 reading a key at position 2 receives the same positional transformation as a query at position 103 reading a key at position 100. Both pairs have $s-t=-3$. Their content vectors can still produce different scores, since $q_t$ and $k_s$ remain inside the expression. RoPE supplies a consistent geometry for the displacement while leaving the model free to decide whether the two pieces of content should interact. If we describe causal distance as the nonnegative number of steps backward, $\Delta=t-s$, the same rotation is written $R_{-\Delta}$.

This placement has two useful consequences. Position changes the direction of a query or key while preserving its norm, and the resulting relative geometry participates directly in the score used to select a source token. Values usually remain unrotated because they carry the information retrieved after that selection has been made. The formula can be evaluated at any integer position, though a trained checkpoint has only learned how to use the phase patterns it encountered within its training context. Much longer contexts can therefore move some frequency pairs into unfamiliar regimes.

Partial RoPE. A query or key head contains many coordinates, and they do not all need to carry an explicit rotary phase. Several modern models reserve one part of each head for position-sensitive comparisons and leave the rest as ordinary content features. If the rotated and unrotated parts are written as superscripts $(r)$ and $(u)$, the split is

\[\widetilde q_t = \begin{bmatrix} R_t q_t^{(r)} \\\\ q_t^{(u)} \end{bmatrix}, \qquad \widetilde k_s = \begin{bmatrix} R_s k_s^{(r)} \\\\ k_s^{(u)} \end{bmatrix}.\]

The dot product then separates into two contributions:

\[\widetilde q_t^\top\widetilde k_s = \left(q_t^{(r)}\right)^\top R_{s-t}k_s^{(r)} + \left(q_t^{(u)}\right)^\top k_s^{(u)}.\]

The first term compares content through the relative rotation $R_{s-t}$. The second compares two unrotated feature vectors. Attention can use both signals in the same score, so a smaller positional slice can still tell the model how two tokens are displaced while the remaining dimensions preserve more room for content features. The size of that slice becomes an architectural choice: increasing it gives position-sensitive interactions more capacity, while decreasing it leaves more of the head untouched by rotary phase.

This split becomes especially useful in multi-head latent attention. MLA needs a large position-independent content path that can be read from a compressed cache, while its smaller rotary path remains explicit. The attention section derives why a position-dependent rotation obstructs the relevant weight absorption. Its tradeoff remains empirical: a smaller rotary slice reduces explicit positional state and gives the score fewer dimensions for position-sensitive interactions.

YaRN. RoPE can calculate a rotation at any integer position, although a checkpoint has learned only from the phase patterns present in its training sequences. YaRN adapts those rotations to a longer context (Peng et al., 2023). Suppose the training length is $L_{\text{train}}=4096$ and the target length is $L_{\text{target}}=32{,}768$. The target contains eight times as many positions, giving the context-length ratio

\[\rho=\frac{L_{\text{target}}}{L_{\text{train}}} =\frac{32768}{4096}=8.\]

Evaluating RoPE at $t/\rho$ maps the longer coordinate range back into the range seen during training: position 32,768 now receives roughly the angles that position 4,096 used before. This operation, called position interpolation, is equivalent to dividing every frequency by $\rho$. It also compresses every local displacement. With $\rho=8$, two tokens eight positions apart receive the angular separation that one token produced during training. A short-wavelength pair with $\lambda=64$ has already repeated many times within 4,096 tokens, so scaling it down sacrifices familiar local resolution. A long-wavelength pair with $\lambda=16{,}384$ has completed only a quarter turn during training; leaving it unchanged through 32,768 tokens would expose the model to phases far beyond that range.

YaRN gives these frequency bands different treatment. It keeps short-wavelength coordinates close to their original frequencies, scales long-wavelength coordinates toward $\omega_i/\rho$, and blends between them:

\[\omega_i^{\text{YaRN}} = \gamma_i\omega_i + (1-\gamma_i)\frac{\omega_i}{\rho}.\]

The weight $\gamma_i$ depends on how many rotations that frequency completed during training, $L_{\text{train}}/\lambda_i$. Values near 1 preserve the original frequency; values near 0 apply full interpolation. YaRN also rescales the attention logits before softmax with an empirically chosen, length-dependent factor, compensating for changes in attention concentration when many more keys are present.

These changes leave the model dimensions untouched, so an existing RoPE checkpoint can use YaRN at inference time, while continuation training lets the model adapt to the new geometry. Static YaRN fixes a target ratio such as $\rho=8$ for every request. At a length of 1,024, a fully interpolated coordinate then reaches only the angle it previously had near position 128. Dynamic YaRN keeps the original frequencies within the 4,096-token training range and increases $\rho$ only as the request grows, reaching eight at 32,768. Directly using the original frequencies at 32,768 gives raw RoPE extrapolation.

The paper measured both inference-only extension and continuation training. In its 8k experiment, inference-only YaRN reached a sliding-window perplexity of 3.49, compared with 3.65 for uniform interpolation. After 400 continuation-training steps, YaRN reached 3.35; uniform interpolation reached 3.34 after 1,000 steps. The result matches the mechanism: preserve familiar local phase changes where possible, rescale the slow rotations that would leave their trained range, and adapt with fewer additional steps.

Relative position bias. RoPE lets distance influence attention through rotations of the query and key. Another approach leaves their geometry unchanged and adds a learned distance preference directly to the attention score. A head could learn, for example, that the immediately preceding token often deserves a small boost, while a token hundreds of positions away should receive no such default preference. The content dot product can still override that prior:

\[\operatorname{score}_{t,s,h} =\alpha q_{t,h}^\top k_{s,h} +b_h(\Delta), \qquad \Delta=t-s,\]

Here $\alpha=1/\sqrt{d_h}$ is the usual attention scale, and $b_h(\Delta)$ supplies the adjustment learned by head $h$ for relative distance $\Delta$. Shifting both tokens forward by the same amount leaves $\Delta$ unchanged, so the preference moves with the pair rather than belonging to an absolute slot.

A direct lookup table also exposes a long-context problem. Representing twice as many exact distances requires twice as many entries. For a frontier model, the raw parameter count may still be manageable; learning those entries and using them outside the training distribution is harder. Each distant bucket receives useful updates only from examples long enough to reach it, and a value learned for distance 900,000 shares no structure with the neighboring value at 900,001. Extending the table after training would simply create untrained entries. Practical designs therefore need some form of sharing, bucketing, interpolation, or a deliberate cutoff.

Inkling combines sharing with a cutoff and lets the distance preference depend on the current query. The layer learns 16 reusable profiles over backward distance. At each query position, every attention head produces 16 mixing weights from the current hidden state:

\[r_{t,h}=W_R^{(h)}h_t\in\mathbb R^{16}.\]

The profiles form a bank $B\in\mathbb R^{16\times D}$, where $D$ is the number of represented backward distances. Column $B_{:,\Delta}$ contains the 16 profile values at distance $\Delta$, and their weighted combination gives the query-specific bias:

\[b_{t,h}(\Delta)=r_{t,h}^\top B_{:,\Delta}, \qquad \Delta=t-s.\]

Each row of $B$ is one learned pattern over distance, while $r_{t,h}$ chooses how to combine those patterns for the current query. The same head can therefore prefer nearby sources for one token and more distant sources for another, while distance still enters as an additive adjustment to the content score.

The represented range is intentionally finite. Inkling’s sliding-window layers cover all 512 positions in their window, and its global layers learn relative bias only for distances from 0 through 1,023. Beyond that cutoff, this positional term contributes zero. A key 2,000 tokens back and one 200,000 tokens back receive no direct distinction from exact distance, although their attention scores can still differ through $q_t^\top k_s$ and through contextual information accumulated by earlier causal layers and short convolutions. Exact distance acts as a learned local prior; remote retrieval becomes primarily content-addressed. The saved table size comes with a clear boundary: a task that needs the exact difference between 20,000 and 200,000 tokens must recover it through other computations.

YaRN and Inkling make different long-context choices. YaRN maintains a rotary coordinate throughout the extended range, while Inkling supplies exact relative distance only inside a bounded neighborhood. Thinking Machines reports that the complete Inkling design performed and extrapolated better than RoPE in its experiments, although the release includes no dedicated positional ablation. The result therefore supports the complete architecture, including its bounded bias, content attention, and short convolutions.

Across the six models, position follows the token mixer and its memory layout. Kimi, DeepSeek, and GLM pair partial RoPE with attention designs that compress or share cached content; Kimi and DeepSeek also extend the rotary path with YaRN. Qwen applies partial RoPE in full-attention layers, while recurrent Gated DeltaNet layers obtain order through causal state updates and short convolution. MAI introduces RoPE through local attention before hidden states reach its global layers. Inkling uses a bounded relative bias and leaves farther retrieval to content. These choices determine what attention can compare and which information must remain in memory, leading directly to attention and caching.

Attention and Memory

Position tells the model where tokens came from. Attention determines which of those tokens each position should read and what information it should take from them. To see the operation clearly, let $x_t\in\mathbb R^{d_{\text{model}}}$ be the hidden state at position $t$, and let $X\in\mathbb R^{T\times d_{\text{model}}}$ stack the hidden states from all $T$ positions entering one attention layer.

An attention head applies three learned projections to every hidden state:

\[Q=XW_Q, \qquad K=XW_K, \qquad V=XW_V.\]

The three projections give each token three roles. The query $q_t$ describes what position $t$ is looking for. A key $k_s$ describes what source position $s$ can be matched by. A value $v_s$ contains the information returned if that source receives attention. Keeping matching and retrieval separate lets two positions match strongly through their query and key while still returning a learned representation through the value.

For position $t$, the head compares $q_t$ with every key it is allowed to read. In a causal decoder, those sources are positions $1$ through $t$:

\[\ell_{t,s} =\frac{q_t^\top k_s}{\sqrt{d_h}}, \qquad s\le t.\]

The dot product is large when the query and key point in similar directions. The division by $\sqrt{d_h}$ keeps its scale controlled. If the coordinates of $q_t$ and $k_s$ have roughly unit variance, their $d_h$ products add to a score whose variance grows with $d_h$. Dividing by $\sqrt{d_h}$ brings that variance back to a roughly constant scale and keeps softmax from becoming excessively sharp merely because the head is wider.

The scores are normalized across the permitted source positions, then used to mix their values:

\[a_{t,s} =\frac{\exp(\ell_{t,s})} {\sum_{j=1}^{t}\exp(\ell_{t,j})}, \qquad o_t=\sum_{s=1}^{t}a_{t,s}v_s.\]

The weights $a_{t,s}$ are nonnegative and sum to one. The output $o_t$ is therefore a weighted mixture of the available value vectors. A large query-key score gives the corresponding value more influence on the new representation at position $t$.

Writing all positions together gives the familiar matrix expression

\[\operatorname{Attention}(Q,K,V) =\operatorname{softmax}\!\left( \frac{QK^\top}{\sqrt{d_h}}+M \right)V,\]

where $M_{t,s}=0$ when $s\le t$ and $M_{t,s}=-\infty$ when $s>t$. Before the mask is applied, $QK^\top$ has shape $T\times T$: one row for every query position and one column for every key position. The causal mask leaves $T(T+1)/2$ permitted pairs, so the arithmetic work of full prompt attention grows as $O(T^2)$. This logical matrix need not be stored in full. FlashAttention visits Q, K, and V in tiles held in fast on-chip memory and maintains the row maximum and normalization factor needed for softmax as it proceeds (Dao et al., 2022). It returns the same exact attention result while avoiding repeated writes and reads of the complete score and weight matrices; the $O(T^2)$ pairwise arithmetic remains.

A multi-head attention layer performs this same operation several times with different learned projections. If there are $n_h$ heads of width $d_h$, each head produces its own output, the outputs are concatenated, and a final projection mixes them. Different heads can therefore learn different notions of relevance while following the same query-key-value calculation.

The KV Cache

Generation runs the same attention operation in two phases. During prefill, the model processes the complete prompt. At every layer it calculates the query, key, and value for all $T$ prompt positions. All $T$ hidden states are needed inside the Transformer, although inference uses only the final logits row to choose the first generated token. During training, the earlier logits rows also predict their shifted next-token targets, as shown in Figure 1.

Now focus on one layer $l$ while the model is generating position $t$. The new token enters with hidden state $h_t^{(l-1)}$ and produces a new query, key, and value:

\[q_t^{(l)}=h_t^{(l-1)}W_Q^{(l)}, \qquad k_t^{(l)}=h_t^{(l-1)}W_K^{(l)}, \qquad v_t^{(l)}=h_t^{(l-1)}W_V^{(l)}.\]

The current position needs the prompt and previously generated tokens to update its representation. At layer $l$, those earlier positions are available as keys and values. The new query compares with the keys to decide which earlier states are relevant, and the corresponding values supply the information that is retrieved:

\[o_t^{(l)} =\operatorname{softmax}\!\left( \frac{q_t^{(l)}\left(K_{\le t}^{(l)}\right)^\top}{\sqrt{d_h}} \right)V_{\le t}^{(l)},\]

where $K_{\le t}^{(l)}=[k_1^{(l)},\ldots,k_t^{(l)}]^\top$ and $V_{\le t}^{(l)}=[v_1^{(l)},\ldots,v_t^{(l)}]^\top$. This output updates the current token before it enters layer $l+1$, whose attention uses a separate set of projections and a separate history.

The formula also shows the lifetime of each tensor. The query side contains only $q_t^{(l)}$. Earlier queries have already produced the outputs for their own positions and never appear in this calculation. The source side contains every earlier key and value. A key-value pair created at position $s$ may be read again by $q_{s+1},q_{s+2}$, and every later query. The current query is consumed once; its key and value become part of the history.

Without saved K/V, the model would have to reconstruct $K_{<t}^{(l)}$ and $V_{<t}^{(l)}$ before it could calculate $o_t^{(l)}$. That means processing the earlier positions through the preceding layers again, then repeating the key and value projections at layer $l$. The same historical results would be regenerated at every decode step. Saving layer inputs alone would still leave the K/V projections to repeat. Attention consumes the projected keys and values directly, so serving systems retain those finished tensors as the KV cache.

Figure 3 places prefill and one decode step in the same tensor layout.

Read the figure from top to bottom. Prefill produces $T$ rows of Q, K, and V. The Q path ends after those rows have produced their attention outputs, while the arrows from K and V fill the layer’s cache. In the decode row, the single new query $q_{T+1}$ multiplies the restored key matrix, producing one row of attention weights. Those weights mix the restored value matrix into one output row. The pale regions are the $T$ cached positions; the darker column in $K^\top$ and darker row in $V$ are the new $k_{T+1}$ and $v_{T+1}$. They join the pale region for the next step, while $q_{T+1}$ is finished.

The cache exists at every decoder layer. For ordinary multi-head attention, a batch of $B$ sequences with retained length $T$, $L$ layers, $n_h$ heads, head width $d_h$, and $b$ bytes per number requires approximately

\[M_{\text{KV}}=2BTLn_hd_hb.\]

The factor of two accounts for both keys and values. With $B=1$, $T=32{,}768$, $L=32$, $n_h=32$, $d_h=128$, and BF16 storage with $b=2$, one sequence requires 16 GiB of KV cache. That memory holds request-specific activations for a single sequence, before accounting for model weights, temporary buffers, or other concurrent requests.

The same cache matters in two closely related ways. Its capacity is the amount of GPU memory that must remain occupied, while its bandwidth cost comes from reading the retained K/V again for every generated token. Under full attention, a new query reads all $T$ cached keys and values at every layer, so longer context increases both the resident state and the cache traffic per token linearly.

Causality also allows reuse across a longer boundary. A later conversation turn begins with the same earlier prefix, and separate requests may share an identical system prompt or document. When the tokens, positions, model weights, and cache representation match, a serving system can reuse those KV blocks and compute only the new suffix. PagedAttention makes variable-length growth and sharing easier to manage by mapping fixed-size logical cache blocks onto noncontiguous physical GPU memory (Kwon et al., 2023). Requests can acquire blocks as they grow, and sequences with a common prefix can point to the same physical blocks, without changing the attention calculation itself.

Compressing the Per-Token Cache

Figure 3 treats the past as $T$ separately addressable K/V rows, with a separate stream for every attention head. The first family of modern changes keeps those $T$ token rows intact and reduces how much each row stores. MQA and GQA share K/V streams across query heads; MLA goes further and replaces the complete per-head K/V tensors with a compact latent representation.

Sharing K/V Heads

In ordinary multi-head attention (MHA), the query, key, and value projections all produce the same number of heads; call that number $H_q$. At source position $s$, the key and value projections therefore produce $H_q$ key vectors and $H_q$ value vectors:

\[h_s \longrightarrow \left\{k_s^{(1)},\ldots,k_s^{(H_q)}\right\}, \left\{v_s^{(1)},\ldots,v_s^{(H_q)}\right\}.\]

Query head $h$ reads the corresponding historical streams $K_{\le t,h}$ and $V_{\le t,h}$. Since these are the persistent tensors from Figure 3, every new token adds $2H_qd_h$ cached values per layer, and those values must be loaded repeatedly as the sequence grows.

Multi-query attention (MQA) addresses this directly (Shazeer, 2019). The query projection still produces $H_q$ query heads, preserving $H_q$ different ways to search the history. The key and value projections each produce only one head. Every earlier token now contributes one key vector and one value vector to the cache, and all query heads read those shared tensors:

\[o_{t,h} =\operatorname{softmax}\!\left( \frac{q_{t,h}K_{\le t}^{\top}}{\sqrt{d_h}} \right)V_{\le t}.\]

The query heads remain separate. Since $q_{t,h}$ differs across heads, they generally assign different weights to the shared keys and produce different mixtures of the shared values. As in ordinary multi-head attention, their $H_q$ outputs are concatenated and passed through $W_O$. MQA keeps those query and output paths while shrinking the persistent source state by a factor of $H_q$.

This aggressive sharing also gives every query head the same projected source representation. Grouped-query attention (GQA) introduces an intermediate choice (Ainslie et al., 2023). The layer keeps $H_q$ query heads and produces $H_{kv}$ K/V heads, where $1<H_{kv}<H_q$:

\[Q\in\mathbb R^{B\times H_q\times T_q\times d_h}, \qquad K,V\in\mathbb R^{B\times H_{kv}\times T_k\times d_h}.\]

The cache stores these smaller K and V tensors directly. When $H_{kv}$ divides $H_q$, each K/V stream serves $H_q/H_{kv}$ query heads. Let $g(h)$ denote the stream assigned to query head $h$. The calculation becomes

\[o_{t,h} =\operatorname{softmax}\!\left( \frac{q_{t,h}K_{\le t,g(h)}^{\top}}{\sqrt{d_h}} \right)V_{\le t,g(h)}.\]

As in MQA, sharing stops at K and V. GQA still produces $H_q$ outputs and follows the usual multi-head output path.

The three designs are now points on the same axis. MHA uses $H_{kv}=H_q$, MQA uses $H_{kv}=1$, and GQA selects an intermediate $H_{kv}$.

Figure 4 shows the head layout for one source position, and the same layout repeats for every token in the cache. MHA, GQA, and MQA all retain the full sequence of $T$ token positions; they store $H_q$, $H_{kv}$, or one K/V vector pair for each position. The change therefore shrinks the per-token head axis while leaving the token axis intact. MQA obtains the smallest cache by asking one source representation to serve every query head. GQA retains several source representations and therefore offers an intermediate capacity-efficiency tradeoff. In the original experiments, it approached MHA quality with inference performance close to MQA, although the useful amount of sharing remains an empirical model choice.

The earlier cache formula now changes in exactly one place:

\[M_{\mathrm{KV}}=2BTLH_{kv}d_hb.\]

Replacing $H_q$ cached heads with $H_{kv}$ reduces both cache capacity and the K/V data read during decoding by the factor $H_q/H_{kv}$. In the 32-query-head example from the previous subsection, MHA required 16 GiB. Keeping eight K/V heads reduces the cache to 4 GiB; keeping one reduces it to 0.5 GiB, with the other dimensions held fixed.

The narrower $W_K$ and $W_V$ projections also use fewer parameters and less projection computation. The main attention calculation changes much less: all $H_q$ query heads still compare with every visible source position and mix a value stream, so the logical QK and AV work remains. The larger decode benefit therefore comes from the cache-capacity and bandwidth reductions quantified above. Inkling and MAI use GQA in their local and global attention layers, while Qwen3.5 uses a gated form in its periodic full-attention layers. Multi-head latent attention pushes the cache question further: can the model store one compact representation for each token instead of several complete K/V heads?

Compressing K/V with MLA

Multi-head latent attention (MLA)(DeepSeek-AI, 2024) starts from the same low-rank down-up idea used by LoRA (Hu et al., 2022). LoRA represents a weight update with a narrow down-projection followed by an up-projection. MLA applies the same pattern to the path that produces keys and values.

In ordinary attention, the hidden state $h_s$ of a source token is projected directly into the key and value for each head:

\[k_{s,h}^{C}=W_h^Kh_s, \qquad v_{s,h}=W_h^Vh_s.\]

MLA replaces these direct projections with two stages. It first sends $h_s$ through a shared learned down-projection, producing a much smaller intermediate vector $c_s^{KV}$. If the hidden state has width $d_{\text{model}}$, this intermediate vector has width $d_c$, where $d_c$ is a chosen bottleneck size. DeepSeek-V2, for example, compresses a 5,120-dimensional hidden state to 512 dimensions.

Call the down-projection matrix $W^{DKV}\in\mathbb R^{d_c\times d_{\text{model}}}$. The superscript $DKV$ means that this is the down-projection shared by the key and value paths:

\[c_s^{KV}=W^{DKV}h_s \in\mathbb R^{d_c}.\]

Each attention head then uses its own up-projections to expand that shared vector into a content key and a value:

\[k_{s,h}^{C}=W_h^{UK}c_s^{KV}, \qquad v_{s,h}=W_h^{UV}c_s^{KV}.\]

The superscript $U$ denotes an up-projection, while $C$ marks the position-independent content part of a key. The complete path is

\[h_s \xrightarrow{\;W^{DKV}\;} c_s^{KV} \xrightarrow{\;W_h^{UK},\,W_h^{UV}\;} \left(k_{s,h}^{C},v_{s,h}\right).\]

Equivalently, each original projection matrix has been factorized:

\[W_h^K=W_h^{UK}W^{DKV}, \qquad W_h^V=W_h^{UV}W^{DKV}.\]

The useful serving opportunity lies in the middle of this factorization. Every head derives its content key and value from the same narrow $c_s^{KV}$, so the decoder can retain that shared vector for each source token and discard the expanded per-head K/V tensors. This stored middle vector is the KV latent. The up-projections still let different heads produce different views of the same latent when the model defines its keys and values.

Taken literally, however, this plan seems to exchange storage for repeated computation. A straightforward decoder would begin every new token by running all historical latents through all of the up-projections:

\[\left\{c_s^{KV}\right\}_{s\le t} \xrightarrow{\;W_h^{UK},\,W_h^{UV}\;} \left\{k_{s,h}^{C},v_{s,h}\right\}_{s\le t,h} \xrightarrow{\;\text{attention}\;} z_t.\]

The history grows with the context, so repeating those projections at every decode step would create substantial computation and temporary tensors. Keeping the expanded results would restore the large KV cache that the latent was intended to remove. MLA becomes practical because its linear maps allow attention to change the order of these operations.

For the moment, set positional encoding aside and consider only the linear query-key calculation. On the key path, the literal order generates $k_{s,h}=W_h^{UK}c_s^{KV}$ for every historical position $s$, then takes its dot product with the current query $q_{t,h}$:

\[q_{t,h}^{\top}k_{s,h} = q_{t,h}^{\top}W_h^{UK}c_s^{KV}.\]

Associativity allows the up-projection to act on the current query instead:

\[q_{t,h}^{\top}W_h^{UK}c_s^{KV} = \left(\left(W_h^{UK}\right)^{\top}q_{t,h}\right)^{\top}c_s^{KV}.\]

Define $\bar q_{t,h}=\left(W_h^{UK}\right)^{\top}q_{t,h}$. This transformed query has width $d_c$, matching the KV latent. The decoder computes it once for the current token and compares it directly with every cached $c_s^{KV}$. One query transformation replaces a separate key up-projection for every historical position. This reassociation is the key side of weight absorption.

The value path follows the same idea. Its literal order expands each historical latent into $v_{s,h}=W_h^{UV}c_s^{KV}$ and then mixes those values with the attention weights $a_{t,s,h}$. Linearity permits the mixture to happen first:

\[z_{t,h}=\sum_s a_{t,s,h}v_{s,h} = \sum_s a_{t,s,h}W_h^{UV}c_s^{KV} = W_h^{UV}\left(\sum_s a_{t,s,h}c_s^{KV}\right).\]

Each head therefore computes one weighted sum of cached latents and applies $W_h^{UV}$ once to the result. The following output projection $W_h^O$ is also linear, so the two matrices can be combined ahead of time:

\[W_h^Oz_{t,h} = \left(W_h^OW_h^{UV}\right) \left(\sum_s a_{t,s,h}c_s^{KV}\right).\]

The key score now reads the latent directly, and the value mixture remains in latent space until the final projection. Per-head K/V still specify the function learned by the layer, while efficient decoding never needs to materialize those tensors across the full history.

Could ordinary MHA apply the same algebra and cache each token’s hidden state $h_s$? In principle, yes. Its key matrix can move to the query side:

\[q_{t,h}^{\top}W_h^Kh_s = \left(\left(W_h^K\right)^{\top}q_{t,h}\right)^{\top}h_s.\]

Its value matrix can likewise be combined with the output projection. This would replace the usual K/V pair with one cached hidden state, although every query head would then attend in the full $d_{\text{model}}$-dimensional hidden space instead of its much narrower $d_h$-dimensional head space. MLA makes the same reassociation practical by first compressing $h_s$ into $c_s^{KV}$, where $d_c\ll d_{\text{model}}$. The latent bottleneck therefore controls both the stored representation and the width of the absorbed attention calculation.

So far, we have deliberately left positional encoding out of the derivation. Weight absorption works because the map between $c_s^{KV}$ and the attention score is linear and independent of the source position. Applying full RoPE after the key up-projection changes that property:

\[\left(R_tq_{t,h}\right)^{\top}R_sW_h^{UK}c_s^{KV} = q_{t,h}^{\top}R_{s-t}W_h^{UK}c_s^{KV}.\]

Absorbing $W_h^{UK}$ into the query would then give

\[\left(\left(W_h^{UK}\right)^{\top}R_{s-t}^{\top}q_{t,h}\right)^{\top}c_s^{KV}.\]

The rotation $R_{s-t}$ depends on the historical position $s$. The decoder could no longer transform the current query once and reuse it across the full cache. It would need a different transformed query for every source position, or it would need to materialize the rotated historical keys.

DeepSeek-V2 preserves the absorbed computation by separating each attention score into a large content lane and a small positional lane. The content query and key, $q^C$ and $k^C$, stay outside RoPE and continue to use the KV latent. The positional query and key, $q^R$ and $k^R$, carry the rotations explicitly. Both positional vectors are necessary because RoPE expresses relative position through the dot product of a rotated query with a rotated key.

DeepSeek-V2 also derives its query slices through a query-side low-rank factorization, primarily to reduce activation memory during training. The current hidden state first produces a query latent $c_t^Q$, and each head expands it into a content query and an initially unrotated positional query. A tilde marks the learned projection before RoPE:

\[c_t^Q=W^{DQ}h_t, \qquad q_{t,h}^{C}=W_h^{UQ}c_t^Q, \qquad \widetilde q_{t,h}^{R}=W_h^{QR}c_t^Q.\]

The query latent belongs only to the current token and is consumed immediately. On the key side, the shared KV latent defines each head’s content key, while a separate projection of the original hidden state produces one positional key:

\[k_{s,h}^{C}=W_h^{UK}c_s^{KV}, \qquad \widetilde k_s^R=W^{KR}h_s.\]

There is no head index on $\widetilde k_s^R$, since every attention head reads the same positional key for source token $s$. RoPE acts only on the two smaller positional vectors:

\[q_{t,h}^{R}=R_t\widetilde q_{t,h}^{R}, \qquad k_s^R=R_s\widetilde k_s^R.\]

Each head then concatenates its content and positional slices:

\[q_{t,h}=\left[q_{t,h}^{C};q_{t,h}^{R}\right], \qquad k_{s,h}=\left[k_{s,h}^{C};k_s^R\right].\]

Figure 5 places these two lanes in one projection graph. The useful question to follow from bottom to top is which intermediate representations disappear after the token has been processed, and which ones remain available to later queries.

The left branch takes $h_t$ through the query latent and produces each head’s $q^C$ and $q^R$. The right branch takes $h_t$ through the KV latent and defines the per-head $k^C$ and $v$, while the narrow center branch produces the shared $k^R$. Content and positional slices are concatenated before Q, K, and V enter multi-head attention. The moving gradient identifies the two representations retained for each source token: $c_s^{KV}$ and $k_s^R$. By following the learned maps in their literal order, Figure 5 gives the projection view of MLA and answers what tensors its parameters define.

Efficient decoding follows an equivalent cache-and-computation view. The earlier weight-absorption identities let each head move $W_h^{UK}$ onto its current query, compare that transformed query directly with the cached $c_s^{KV}$ vectors, mix those same latent vectors, and apply $W_h^{UV}$ after the mixture. The expanded per-head content keys and values therefore never need to be materialized for the history. Core attention sees one shared latent K/V sequence read by many query heads, which gives it the data layout of MQA. The MQA label describes this evaluation order; the layer remains MLA and produces the same result as the projection view. The smaller $q^R$/$k^R$ lane stays explicit because its position-dependent rotations cannot be absorbed in the same way. Figure 7 will draw this second view while adding the sparse indexer.

Implementations often call the content slices $q_{\mathrm{nope}}$ and $k_{\mathrm{nope}}$, and the rotary slices $q_{\mathrm{pe}}$ and $k_{\mathrm{pe}}$. Concatenation makes the score split into two independent dot products:

\[q_{t,h}^{\top}k_{s,h} = \left(q_{t,h}^{C}\right)^{\top}k_{s,h}^{C} + \left(q_{t,h}^{R}\right)^{\top}k_s^R = \left(\bar q_{t,h}^{C}\right)^{\top}c_s^{KV} + \left(R_t\widetilde q_{t,h}^{R}\right)^{\top} \left(R_s\widetilde k_s^R\right).\]

The first term is the absorbed content comparison. The second contributes relative position through the smaller rotary slice. Modern implementations often describe this as partial RoPE; the DeepSeek-V2 report calls this particular separation decoupled RoPE.

The asymmetry between Q and K is deliberate. Every head receives its own $\widetilde q_{t,h}^{R}$ and can learn a different positional preference. A historical token stores one shared $k_s^R$, which every query head compares with its own positional vector. The design therefore keeps head-specific positional scores while adding only one small positional vector to the cache.

Each source token ultimately retains

\[\operatorname{Cache}(s)=\left[c_s^{KV},k_s^R\right],\]

with width $d_c+d_R$. The large content lane supports weight absorption through the shared KV latent, and the small positional lane keeps RoPE outside that absorbed path. DeepSeek introduced this layout in DeepSeek-V2, and the same cache design appears in Kimi and GLM models.

The absorbed evaluation order also shapes how Kimi K2 handles a separate training problem. While scaling the Muon optimizer, the team observed attention logits grow beyond 1,000 in a mid-scale run, a failure that appeared more frequently with Muon than with AdamW in their experiments (Kimi Team, 2025). Scores at this scale can make softmax nearly one-hot, cause numerical instability, and produce loss spikes or training divergence.

This may seem surprising because scaled dot-product attention already divides every query-key score by $\sqrt{d_h}$. That denominator addresses a particular source of growth. The raw dot product

\[q^\top k=\sum_{i=1}^{d_h}q_i k_i\]

adds one coordinate product for every dimension in the head. If those products have roughly constant variance and are weakly correlated, the variance of their sum grows in proportion to $d_h$, and its standard deviation grows like $\sqrt{d_h}$. Dividing by $\sqrt{d_h}$ therefore keeps the typical score scale comparable when the head width changes.

Training can enlarge the coordinates themselves while $d_h$ stays fixed. If the learned projections make every coordinate of both $q$ and $k$ twice as large, every product $q_i k_i$ becomes four times as large. The denominator remains $\sqrt{d_h}$, so the normalized attention logit also becomes four times as large. Equivalently,

\[q^\top k=\lVert q\rVert\lVert k\rVert\cos\theta,\]

and the two vector norms can grow while their angular match remains unchanged. The original Transformer and GPT-1 used the width correction alone. That fixed factor can be sufficient while the training dynamics keep the projection norms in a stable range; it provides no bound once those norms begin to grow. QK-Norm adds a second form of control for training regimes where learned vector growth becomes a problem: it normalizes the projected query and key heads before their attention scores are formed, often with LayerNorm or RMSNorm.

MLA affects how this second control can be implemented. During training, the expanded per-head keys may exist as temporary activations, so applying QK-Norm there is straightforward. Inference must reproduce the same normalized attention function, while absorbed MLA compares the transformed query directly with each cached KV latent and never constructs the complete historical content keys. QK-Norm is data-dependent and nonlinear, which means it cannot be moved onto the current query through the same matrix reassociation used for $W_h^{UK}$. Reproducing it during decoding would require expanding the historical keys from the latent cache or retaining them after they are produced. Either choice gives up an important part of the efficient MLA path.

Kimi therefore introduced QK-Clip to preserve absorbed inference while controlling the instability observed during Muon training. For each head in each layer, the training forward records one scalar $S_{\max}^{h}$: the largest valid pre-softmax score across the query-key token pairs in the current batch. The current forward is left unchanged. After the optimizer update, a head whose recorded maximum exceeds a threshold $\tau$ receives the scale

\[\gamma_h=\min\left(1,\frac{\tau}{S_{\max}^{h}}\right).\]

The head-specific content query and key projections are each multiplied by $\sqrt{\gamma_h}$, so their dot product is multiplied by $\gamma_h$. The rotary score needs slightly different treatment: its query projection belongs to one head, while its key is shared across all heads. Kimi therefore multiplies the head-specific rotary query projection by $\gamma_h$ and leaves the shared rotary key unchanged. The next forward pass produces a smaller score for the offending head without adding normalization to the inference graph or changing the latent cache. QK-Clip leaves ordinary variation in query and key lengths intact and intervenes when the observed logit maximum crosses the threshold.

Choosing Which Tokens to Read

MLA reduces the representation stored for each token while keeping every historical position separately addressable. A dense MLA layer still asks every new query to read all of those positions. Long-context models can make a second, independent change: reduce the set of source positions that attention reads. Figure 6 shows this set directly. Each row belongs to one query position, each column belongs to one key position, and every filled square is a query-key pair computed by the main attention operation.

Global attention. The first matrix is the familiar causal pattern. Query $t$ reads every key up to its own position,

\[\mathcal S_t^{\mathrm{global}}=\{1,\ldots,t\}.\]

This produces a staircase of filled cells below the main diagonal. During decoding, the model computes only the newest row, and that row grows by one square after each generated token. Every historical position remains separately addressable.

Sliding-window attention. A sliding-window layer restricts each row to the most recent $W$ keys,

\[\mathcal S_t^{\mathrm{local}} = \{\max(1,t-W+1),\ldots,t\}.\]

The Sliding Window matrix in the top row of Figure 6 describes direct access within one layer. Stacking local layers expands the set of input tokens that can influence a later hidden state, even though the direct window in every layer remains the same size. Suppose $W=3$. At layer 2, state $h_t^{(2)}$ directly attends to $h_{t-2}^{(1)}$, $h_{t-1}^{(1)}$, and $h_t^{(1)}$. The first of those states, $h_{t-2}^{(1)}$, was itself computed from input positions $t-4$, $t-3$, and $t-2$. Information from position $t-4$ can therefore reach $h_t^{(2)}$ through $h_{t-2}^{(1)}$, although layer 2 never computes an attention score directly between positions $t$ and $t-4$.

This distinction explains the two lower panels. The dark cells in the lower-left panel are positions read directly in the current layer; the lighter cells are positions whose information can arrive through hidden states produced by lower layers. The lower-right panel unfolds the same relay by input position. Each additional local layer can extend the earliest reachable input by another $W-1$ positions. With window width $W$ repeated across $L$ layers, the widest span that can influence one output is

\[R_L=1+L(W-1)\]

positions. With $W=3$, coverage grows from one position at the input to three after layer 1, five after layer 2, and seven after layer 3. The final query still assigns attention weights to three states in the preceding layer. Information from earlier positions has been summarized into those intermediate states, so this indirect path is less precise than giving the final query direct access to every earlier token.

The window changes the number of pairs from roughly $T^2/2$ to roughly $TW$ during prefill, and a decode step reads at most $W$ positions in that layer. Cache capacity becomes bounded when the runtime also discards entries that no future query can reach. Keeping all old entries while applying a local mask saves attention work and leaves cache size unchanged.

Periodic global attention. A model can combine the first two patterns across depth. Inkling and MAI-Thinking-1 use groups of five 512-token local layers followed by one global layer. The local layers move nearby information forward at bounded cost. The global layer then gives every position a fresh direct comparison with the full prefix. It can recover a specific remote detail that may have been weakened during several local hops, and its output carries that global information into the next local group.

Attention sinks. The third matrix keeps the initial key visible beside the recent window. This pattern comes from a property of softmax: every head must distribute a total probability mass of one across the keys it can see. When no visible value is useful, some pretrained models place excess probability on the first few tokens. Those tokens are available to nearly every later query during causal training and become stable destinations for this probability.

A rolling cache that removes those initial entries also removes the destinations the model has learned to use. StreamingLLM retains a few initial K/V entries together with the recent window, producing the vertical column and diagonal band in Figure 6 (Xiao et al., 2023). Four initial tokens were sufficient for the models evaluated in that work. The retained tokens stabilize attention; content discarded from the middle of the sequence remains unavailable.

DeepSeek-V4 builds a sink directly into each attention head. A learned logit $z_h$ joins the softmax denominator and has no associated value:

\[a_{t,s} = \frac{\exp(\ell_{t,s})} {\exp(z_h)+\sum_{j\in\mathcal S_t}\exp(\ell_{t,j})}.\]

Probability assigned to the sink disappears before value mixing. A head can therefore reduce the total contribution from real tokens without assigning that probability to an ordinary content token.

DeepSeek Sparse Attention (DSA). A sliding-window layer decides what each query may read with a fixed distance rule. DSA instead lets the query choose a small set of positions according to their content, including positions far outside a local window. The fourth matrix in Figure 6 illustrates the result: every row may select a different set of columns.

The most direct way to find those positions would be to run ordinary MLA over the entire history, inspect the resulting attention scores, and keep the $k$ largest ones. This would tell us exactly which positions the main attention layer prefers. It would also be too late to save any work: producing those scores already requires the model to read the full MLA cache and compare the query with every historical token. DSA therefore needs a cheaper first pass that can scan the full history and choose promising positions before MLA performs its larger calculation.

Figure 7 returns to the cache-and-computation view introduced after Figure 5. Its core is labeled Multi-Query Attention because absorption has removed the explicit per-head content K/V tensors from the historical side. DSA still uses the same MLA layer; this evaluation order exposes the cached tensors on which Top-k selection acts. Both computations begin from the same input hidden state. Along the neutral path, $h_t$ produces the query latent $c_t^Q$ and a new MLA cache entry $[c_t^{KV};k_t^R]$. The latent $c_t^Q$ expands into the main MLA queries, while $c_t^{KV}$ supplies the content keys and values used by core attention. Along the accented path, the same $c_t^Q$ produces the indexer queries, and $h_t$ produces a new index key and the head weights. The indexer chooses token positions, and those positions determine which complete MLA cache entries reach core attention.

That first pass is a learned module called the Lightning Indexer (DeepSeek-AI, 2025). Recall the division of labor inside ordinary attention: Q and K decide how strongly two positions match, while V carries the information retrieved from the chosen positions. At this first stage, DSA only needs to decide which past positions look promising. It does not yet need to retrieve and mix their values. The indexer can therefore use a smaller Q/K-style comparison and omit the value path entirely.

For every source token $s$, the indexer projects its hidden state $h_s$ into one narrow vector $k_s^I$. Once token $s$ has been processed, this vector is kept so that later query tokens can compare against it. The accumulated sequence ${k_1^I,\ldots,k_t^I}$ is what Figure 7 calls the cached index keys. In DeepSeek-V3.2-Exp, it has shape $[T,128]$: one 128-dimensional vector per historical token, with no head axis. For the current token, the official implementation reuses the MLA query latent $c_t^Q$ to produce 64 separate index queries, and projects $h_t$ into one scalar weight for each query:

\[Q_t^I=\operatorname{reshape}\!\left(W_Q^I c_t^Q\right) \in\mathbb R^{H_I\times d_I}, \qquad k_s^I=\operatorname{Norm}\!\left(W_K^I h_s\right) \in\mathbb R^{d_I}, \qquad w_t^I=W_w^Ih_t \in\mathbb R^{H_I}.\]

The $H_I$ rows of $Q_t^I$ are the indexer heads. Multiple query vectors let the current token search the same history in several learned directions. The paper does not claim that multiple heads are mathematically necessary; they simply give this small scorer more capacity than a single dot product. These query vectors are needed only for the current token, so the model produces them once, uses them for the current search, and then discards them. The index keys must remain available because every future query will scan them. Giving each indexer head its own historical key would multiply the persistent index cache and the amount of data read during every scan. DSA keeps several query heads while sharing one index key across those heads for each historical position.

These heads initially give us several match values for the same source token. The top-$k$ selector needs one score per source position, so DSA combines the head-level matches as

\[I_{t,s} = \sum_{j=1}^{H_I} w_{t,j}^I \operatorname{ReLU}\!\left( \left(q_{t,j}^I\right)^\top k_s^I \right).\]

Here, $j$ identifies an indexer head. The dot product measures how strongly source token $s$ matches that head’s current search direction. The scalar $w_{t,j}^I$ is also produced from the current hidden state, so the model can change how much each search direction matters from one query token to the next. These weights combine the several head-level matches into the single score required to rank source token $s$. Ordinary attention can keep its heads separate because each head produces its own value mixture and the resulting vectors are combined later. The indexer must return one shared list of token positions, so its heads have to be combined before the top-$k$ operation.

ReLU is applied to each dot product before the heads are combined. A negative match becomes zero, while a positive match retains its magnitude. This gives a simple intuition in which every head acts as a positive-match detector and $w_{t,j}^I$ decides how much that detector contributes. The DeepSeek report gives a more practical reason for this exact activation: ReLU is inexpensive and was chosen for throughput. DeepSeek also applies RoPE to 64 of the 128 dimensions in each index query and key, allowing this preliminary ranking to use both content and relative position.

The indexer repeats this small calculation for every source position allowed by the causal mask, then returns the $k$ positions with the largest $I_{t,s}$. At this point, the index scores have completed their job. They decide which positions survive and are not reused as the attention logits of the main layer. Core MLA gathers the full $\left[c_s^{KV},k_s^R\right]$ cache entries at the selected positions and performs its normal calculation on this shorter sequence. Each main attention head computes new content and rotary logits, applies softmax across the selected positions, mixes their latent values, and contributes to the final output projection. A position missed by the indexer is unavailable to core attention in that layer.

The official DeepSeek-V3.2-Exp configuration makes the scale difference concrete. Its hidden states have width 7,168. Main MLA uses 128 attention heads, with 128 content dimensions and 64 rotary dimensions in each logical Q/K head, and retains a 512-dimensional KV latent plus a 64-dimensional rotary key for every token. The Lightning Indexer uses 64 query heads of width 128:

\[Q_t^I\in\mathbb R^{64\times128}, \qquad w_t^I\in\mathbb R^{64}, \qquad k_s^I\in\mathbb R^{128}.\]

Every source token therefore adds one shared 128-dimensional index key, stored in FP8 in the official inference implementation, and the indexer eventually keeps 2,048 positions. These dimensions also show that “lightning” is relative. For one query-source pair, the indexer compares $64\times128=8{,}192$ pairs of scalar coordinates. Computing the QK dot products for all 128 main MLA heads, each with 128 content dimensions and 64 rotary dimensions, would involve $128\times192=24{,}576$ such coordinate pairs. The first pass is therefore about three times smaller at this part of the calculation. It saves more work by skipping value mixing across the full history, reading one narrow FP8 index key per token, and fetching the larger MLA cache entries only after the 2,048 positions have been chosen.

A final question remains: how does this smaller scorer learn which tokens the full attention layer would have preferred? DeepSeek begins with a dense warm-up stage. For each query position, it sums the main attention scores across heads and L1-normalizes them over the source positions, producing a target distribution $p_{t,:}$. It then trains the indexer with

\[\mathcal L_I = \sum_t D_{\mathrm{KL}}\!\left( p_{t,:} \,\middle\|\, \operatorname{Softmax}(I_{t,:}) \right).\]

The indexer is trained to reproduce this distribution over source positions. Vocabulary logits do not enter the KL objective above; they continue to train the model through the ordinary next-token loss. After warm-up, DeepSeek enables top-$k$ selection and continues training the model while core MLA can see only the selected positions. The indexer also keeps an alignment objective based on the main attention scores available on that selected set. This stage gives the rest of the model a chance to adapt to the sparse retrieval pattern. Retrieval can still fail, and a missed position cannot contribute within that layer.

We can now state exactly which work DSA removes and which work remains. For a sequence of length $T$, core attention changes from $O(T^2)$ to $O(Tk)$. The indexer still compares every causal query-source pair during prefill, so its scan remains $O(T^2)$. During decoding, it scans $T$ narrow index keys, chooses $k$ positions, and core attention reads only those $k$ larger cache entries. The full-history scan still exists, though each comparison is cheaper; the expensive attention calculation and the larger cache reads are limited to the selected positions.

GLM-5.2 uses the same two-stage mechanism and also selects 2,048 positions (GLM-5 Team, 2026). Once core attention has been reduced to $k$ positions, the full-history index scan itself becomes a large fraction of the remaining work. At the same time, neighboring layers often select substantially overlapping token positions. IndexCache measured this overlap and developed sharing-aware training around it (Bai et al., 2026). The observation suggests that nearby layers frequently agree on where useful information is located, even when they use that information differently.

IndexShare turns that overlap into a serving optimization. In GLM-5.2’s regular four-layer groups, one layer runs its Lightning Indexer and the next three layers reuse the resulting top-$k$ token positions. Only those addresses are shared. Each layer still forms its own MLA queries, retrieves its own layer-specific cache entries at the selected positions, computes new attention logits and softmax weights, and mixes its own values. Reuse therefore removes three out of four full-history indexer scans while preserving a different attention calculation in every layer. A shared layer cannot recover a position that the preceding full-indexer layer omitted, so one missed candidate can remain unavailable throughout the group.

Compressing Token History into Fewer Entries

DSA reduces how many source positions enter the expensive attention calculation, although the history it searches still grows one token at a time. Each source token contributes an MLA cache entry and an index key, and every new query scans all of those index keys before choosing a smaller set. DeepSeek-V4 asks a more aggressive question: can the remote history itself contain fewer entries (DeepSeek-AI, 2026)?

Before compressing several tokens together, V4 simplifies what one source entry contains. In ordinary attention, a key is the address used for matching and a value is the payload returned after the match. V4 treats the features that make a source entry retrievable as the same features worth returning from it, and represents both roles with one vector $c_s$. Query head $i$ compares its query with $c_s$, then uses the resulting weight to mix that same $c_s$:

\[\alpha_{t,i,s} = \operatorname{Softmax}_s\!\left( \frac{q_{t,i}^{\mathsf T}c_s}{\sqrt{d_h}} \right), \qquad o_{t,i}=\sum_s\alpha_{t,i,s}c_s.\]

This is Shared Key-Value MQA. Multiple query heads still ask different questions and therefore produce different attention weights, while every head reads one shared source representation. Ordinary MQA also shares its source-side tensors across query heads, though it retains one K tensor and one separate V tensor. Shared K=V collapses those two source roles into a single representation.

The vector $c_s$ also differs from the MLA latent introduced earlier. MLA stores an intermediate code $c_s^{KV}$ and uses separate up-projections to produce each head’s K and V; the widths of those projected vectors are independent choices. V4 sends $c_s$ directly into core attention as both K and V. It saves the separate value representation and also gives up MLA’s head-specific value projections. The query heads can vary where they read, though the source features being mixed are shared across all heads.

After attention, each of V4-Pro’s 128 query heads returns one 512-dimensional vector. Concatenating them all at once would produce a 65,536-dimensional vector before the ordinary output projection. V4 factorizes this large projection by dividing the heads into a fixed number of groups. Here $g$ is the group index, ranging from $1$ to $G$; the architecture fixes $G=16$, so each group contains eight heads. The eight outputs in group $g$ are concatenated into 4,096 dimensions and passed through that group’s own $W_g^{OA}$, which compresses them to 1,024 dimensions. There are therefore 16 separately learned $W_g^{OA}$ matrices.

The 16 smaller group outputs are then concatenated into a 16,384-dimensional vector. A final shared matrix $W^{OB}$ maps it to the model’s fixed hidden width of 7,168, allowing the result to re-enter the residual stream. Training learns the entries of these matrices, while the number of groups and their widths remain architecture hyperparameters. This two-stage factorization reduces the cost of the output projection, and $W^{OB}$ still allows information from different groups to mix.

The next step reduces the number of source entries. Begin with the simpler case in which four old token states must become one remote-memory entry. Taking their average would give every token equal influence. Learning one scalar weight per token would let the model favor a more useful token, while forcing every feature of the summary to use the same four weights. A block may need its entity feature from one token, its syntactic feature from another, and a numerical feature from a third.

V4 therefore projects each hidden state $h_s$ into a candidate vector $C_s$ and a vector of compression logits $Z_s$:

\[C_s=h_sW^{KV}, \qquad Z_s=h_sW^Z.\]

$C_s$ contains what token $s$ could contribute. $Z_s$ controls which coordinates it is likely to contribute. A learned bias $B_s$ represents the token’s slot within the block. For every feature coordinate $r$, the tokens compete through a separate softmax:

\[A_{s,r} = \operatorname{Softmax}_{s\in\mathcal B}(Z_{s,r}+B_{s,r}), \qquad C^{\mathrm{comp}}_r = \sum_{s\in\mathcal B}A_{s,r}C_{s,r}.\]

For the four-token example, the four weights for coordinate $r$ sum to one. Another coordinate receives another set of four weights. The resulting $C^{\mathrm{comp}}$ is one learned memory of the block, used directly as both key and value. Future queries can retrieve features that survived this pooling, while the original token entries are no longer individually available through the compressed path.

Compressed Sparse Attention (CSA) adds an overlap to this basic compressor. Divide the source positions into $m$-token chunks $\mathcal T_0,\mathcal T_1,\ldots$, where every position $j$ still contributes its own hidden state $h_j$. To construct entry $i$, CSA reads the target chunk $\mathcal T_i$ through an $a$ projection and the preceding chunk $\mathcal T_{i-1}$ through a separate $b$ projection:

\[C_j^a=h_jW^{aKV}, \qquad Z_j^a=h_jW^{aZ}, \qquad j\in \mathcal T_i,\] \[C_j^b=h_jW^{bKV}, \qquad Z_j^b=h_jW^{bZ}, \qquad j\in \mathcal T_{i-1}.\]

$Z^a$ and $Z^b$ are learned compression logits, one value for every source position and candidate-vector coordinate. The compressor adds learned within-block position biases $B^a$ and $B^b$, then normalizes the $2m$ source positions separately for every coordinate. Call the resulting feature-wise weights $S^a$ and $S^b$:

\[[S^a;S^b] = \operatorname{Softmax}_{\text{source position}} ([Z^a+B^a;Z^b+B^b]).\]

For every feature coordinate, the corresponding $2m$ weights sum to one. The final compressed entry combines the two candidate stacks with those feature-wise weights:

\[C_i^{\mathrm{comp}} = \sum_{j\in\mathcal T_{i-1}}S_j^b\odot C_j^b + \sum_{j\in\mathcal T_i}S_j^a\odot C_j^a.\]

The elementwise products make the pooling feature-specific: coordinate $r$ of $S_j$ controls how much coordinate $r$ of candidate $C_j$ contributes. Only the resulting $C_i^{\mathrm{comp}}$ is retained as the cached entry. With $m=4$, consecutive entries have the following source ranges:

\[C_i^{\mathrm{comp}}\leftarrow \mathcal T_{i-1}^{b}+\mathcal T_i^{a}, \qquad C_{i+1}^{\mathrm{comp}}\leftarrow \mathcal T_i^{b}+\mathcal T_{i+1}^{a}.\]

Each entry can therefore draw from eight source tokens, and the compressor advances by four tokens before producing the next entry. The hidden states in the shared chunk $\mathcal T_i$ are projected differently in its two roles. One new entry for every four new tokens still gives a sequence of roughly $T/4$ entries.

CSA runs a second compressor over the same source blocks for the Lightning Indexer. It uses the same overlapping, feature-wise pooling rule with its own learned projections and produces one narrower index key $K_i^{I\mathrm{Comp}}$ for every main entry $C_i^{\mathrm{comp}}$. The two sequences are aligned by $i$: the Indexer scans the smaller keys, and a selected index retrieves the corresponding compressed K=V entry. In the V4-Pro implementation, the main entries are 512-dimensional and the index keys are 128-dimensional. This differs from MLA’s per-token latent: block compression combines several source positions into one entry, and the index key is used directly for retrieval without an up-projection into formal K/V.

Figure 8 places the two aligned compressors inside the complete attention path. Their matching shapes indicate that both use the feature-wise pooling rule described above, although they have separate parameters and output widths. Each main compressor output $C_i^{\mathrm{comp}}$ is already one Shared K=V entry, so the stack labeled compressed K=V entries is simply the sequence ${C_i^{\mathrm{comp}}}$. The parallel output $K_i^{I\mathrm{Comp}}$ is its narrower retrieval key, and the two stacks remain aligned by $i$. An exact local branch joins the selected remote entries before Shared K=V attention.

The compressed sequence can still be long. One query token produces several indexer query heads, which jointly assign one score to each eligible compressed entry. The Lightning Indexer keeps the top 1,024 entries for core attention in V4-Pro. Token-level DSA selected individual MLA cache entries; CSA selects learned block memories. Both the indexer scan and the core-attention read now operate on the shorter compressed sequence.

Compression alone leaves a gap around the current position. A block summary becomes available only after the block is complete, and causality prevents a query from reading a summary that contains later tokens from its own block. Recent wording and syntax also benefit from exact token-level access. Every CSA layer therefore carries a second, uncompressed sliding-window path. The selected remote summaries and the recent exact entries are concatenated before Shared K=V attention.

Heavily Compressed Attention (HCA) changes three choices at once. CSA uses 4:1 overlapping compression with two $a/b$ compressor branches, then relies on an indexer to retrieve a query-specific top-$k$ subset. HCA uses 128:1 non-overlapping compression with a single compressor branch, then lets each query attend densely to every compressed entry.

HCA makes dense remote attention affordable by shrinking the remote sequence much more aggressively. A history of 32,768 tokens becomes 256 remote entries, so scanning all of them is already cheap enough to remove the indexer. The tradeoff is coarser remote memory. HCA therefore keeps the same exact local window used by CSA for recent tokens and the unfinished block.

V4-Pro uses both mechanisms within the same stack. Its 61 main layers begin with HCA, HCA, then alternate CSA, HCA through the remaining layers. This gives 31 HCA layers and 30 CSA layers. As the representation moves through depth, it repeatedly passes between cheap, coarse remote coverage and finer, query-dependent retrieval, while every layer retains an exact 128-token local window.

A shared implementation detail across CSA and HCA is how position enters the final attention read. The learned slot bias inside either compressor identifies where a token lies within its block, while partial RoPE tells a later query how far away the resulting local or compressed entry is. Because the rotated Shared K=V entry also serves as the value, the attention output carries the source entry’s rotary phase. V4 applies the inverse query-position rotation to the corresponding output slice. A contribution from source position $s$ to query position $t$ then carries $R_{-t}R_s=R_{s-t}$, which restores a relative displacement before the grouped output projection. QK-Norm and the learned attention sink described earlier are applied at this final attention read as well.

These choices also produce different cache states. Every layer retains a bounded exact local window and the partial block currently being accumulated. HCA adds one remote Shared K=V entry per completed 128-token block. CSA adds finer remote entries, a separate compressed index-key sequence, and the overlap required by its two-branch compressor.

Compressing Token History into a Fixed-Size State

CSA and HCA replace groups of tokens with a shorter sequence of memory entries, although that sequence still grows as more blocks are completed. Linear attention goes one step further and folds the entire history into a recurrent state whose shape is fixed by the head dimensions. Qwen3.5 uses Gated DeltaNet for this role in most layers, then periodically returns to global softmax attention. The path from ordinary attention to that design begins with the cost of reading a token list.

Linear attention. At position $t$, ordinary causal attention compares $q_t$ with $k_1,\ldots,k_t$. That row contains $t$ query-key comparisons. Processing all $T$ positions therefore evaluates

\[1+2+\cdots+T = \frac{T(T+1)}{2} = O(T^2)\]

comparisons. Prefill performs many of them in parallel, although the arithmetic remains quadratic. During generation, a KV cache avoids recomputing old keys and values, while each new query still scans a history whose length grows with $t$. The left half of Figure 9 shows this growing query path.

To avoid that scan, the layer needs to combine the history before the next query arrives. Temporarily use the dot product itself as the score and omit normalization. The output can then be regrouped as

\[\begin{aligned} o_t &= \sum_{i\le t}v_i(k_i^\top q_t) \\\\ &= \left(\sum_{i\le t}v_ik_i^\top\right)q_t \\\\ &= S_tq_t, \end{aligned}\]

where

\[S_t = \sum_{i\le t}v_ik_i^\top = S_{t-1}+v_tk_t^\top.\]

Each key-value pair updates the same matrix $S_t$. A later query reads that matrix directly, so it never needs to revisit the individual keys. For fixed key and value dimensions, the shape of $S_t$ stays constant as the sequence grows. Updating and reading it take constant work with respect to the history length, giving $O(T)$ work across $T$ recurrent steps. This linear growth in sequence length gives linear attention its name. The right half of Figure 9 depicts this fixed-state path.

This equality also shows what has happened to the history. The tokens still influence the output, although their contributions have been mixed inside $S_t$. A future query can recover only what the shared matrix preserves. The equality itself is exact for this dot-product score.

The same construction works for a broader class of similarity rules. Transform each query and key into $r$ features and define

\[\kappa(q,k) = \phi(q)^\top\phi(k), \qquad \phi(x)\in\mathbb R^r.\]

The state and its read become

\[S_t = \sum_{i\le t}v_i\phi(k_i)^\top, \qquad o_t=S_t\phi(q_t).\]

The $r$ coordinates of $\phi$ can be viewed as $r$ shared memory channels. A key controls how strongly its value is written into each column of $S_t$, and a query controls how those columns are combined during the read. Every token writes into the same $r$ channels. After any number of tokens, the history can affect a future query only through those channels.

For example, imagine a state with two channels. If three keys have feature vectors $(1,0)$, $(0,1)$, and $(1,1)$, the two columns of the state contain $v_1+v_3$ and $v_2+v_3$. A later query can mix these two columns in different proportions, although it cannot recover all three values independently. This is where the fixed-size history compression enters. Writing the same computation as an explicit sum over tokens would give the same result and the same information restriction.

So far, the coefficients have no requirement to be positive or sum to one. A normalized form can retain the weighted-average structure of ordinary attention by using a nonnegative $\phi$:

\[o_t = \frac{\sum_{i\le t}\left[\phi(q_t)^\top\phi(k_i)\right]v_i} {\sum_{i\le t}\phi(q_t)^\top\phi(k_i)}.\]

The numerator is already summarized by $S_t$. The denominator asks for the total score assigned by the current query, so the layer keeps a second state

\[z_t = \sum_{i\le t}\phi(k_i) = z_{t-1}+\phi(k_t).\]

The normalized read is therefore

\[o_t = \frac{S_t\phi(q_t)} {z_t^\top\phi(q_t)}.\]

The vector $z_t$ records how much key mass has accumulated in each feature channel. The query uses the same channel weights to read both the value sum in $S_t$ and the total score in $z_t$. Dividing the two makes the effective weights sum to one.

This leads to the central design question: what should $\phi$ be? Softmax attention uses the positive score

\[\exp(q^\top k).\]

Matching softmax exactly would require a feature map satisfying

\[\phi(q)^\top\phi(k) = \exp(q^\top k)\]

for every possible query and key. In one scalar dimension, the Taylor expansion reveals the required features:

\[\exp(qk) = 1+qk+\frac{q^2k^2}{2!}+\frac{q^3k^3}{3!}+\cdots = \phi(q)^\top\phi(k),\]

with

\[\phi(x) = \left[ 1, x, \frac{x^2}{\sqrt{2!}}, \frac{x^3}{\sqrt{3!}}, \ldots \right]^\top.\]

The exact map contains every polynomial degree and therefore has infinitely many coordinates. With vector queries and keys, each degree also creates cross-coordinate terms. An exact fixed-width state for the exponential score is unavailable over the full continuous input space.

Practical methods commonly follow one of two routes. One route uses a finite feature map designed to approximate the exponential score over the queries and keys the model is likely to encounter. Performer uses positive random features for this purpose (Choromanski et al., 2021). Increasing the feature width $r$ can improve the approximation, while the state size and per-token work grow with $r$:

\[S_t\in\mathbb R^{d_v\times r}, \qquad \text{state size}=O(d_vr), \qquad \text{work over }T\text{ tokens}=O(Td_vr).\]

For any chosen finite $\phi$, regrouping the token sum into $S_t$ and $z_t$ is exact. Approximation enters when that finite feature map is used in place of the exponential score.

The other route defines a different positive, factorized score and trains the model to use it. The original Linear Transformer used the element-wise map

\[\phi(x)=\operatorname{ELU}(x)+1,\]

which is inexpensive, positive, and retains a nonzero gradient for negative inputs (Katharopoulos et al., 2020). Positivity and normalization guarantee a valid weighted average, although they leave the relative weights unspecified. Scores $(e^2,1)$ normalize to approximately $(0.881,0.119)$, while scores $(2,1)$ normalize to $(0.667,0.333)$. Both pairs are positive and sum to one after normalization, yet they retrieve the two values with different strengths. A model trained from scratch can adapt its Q/K projections to the chosen similarity rule.

Later recurrent mixers often use the fixed matrix directly and omit the explicit normalization state $z_t$. Their learned projection path can also transform and normalize Q and K before the recurrence. The 2024 parallel DeltaNet implementation, for example, applies SiLU followed by $\ell_2$ normalization to Q and K (Yang et al., 2024). Writing those final transformed vectors simply as $q_t$ and $k_t$, the recurrence returns to

\[S_t=S_{t-1}+v_tk_t^\top, \qquad o_t=S_tq_t.\]

Without $z_t$, this mixer no longer forces its coefficients to be positive or sum to one. Q/K normalization, state decay, output normalization, and learned gates help control the scale of the resulting recurrent state.

DeltaNet. Recall the roles of Q, K, and V in ordinary attention. A query $q$ describes what the current token is looking for. Each key $k_i$ acts as an address describing which queries should match that item, and its value $v_i$ carries the information to retrieve. Ignoring softmax normalization for a moment, the dot product

\[a_i=k_i^\top q\]

measures how strongly the query matches the $i$th key. Multiplying by the corresponding value gives $a_iv_i$, so a strong match returns more of that value. Ordinary attention performs this operation separately for every key-value pair and then adds the results.

The recurrent state performs the same read after combining K and V in advance. For one key-value pair,

\[(v_ik_i^\top)q = v_i(k_i^\top q) = a_iv_i.\]

The outer product $v_ik_i^\top$ is therefore a small linear map that implements one retrieval rule: compare an incoming query with $k_i$, then return $v_i$ scaled by that match. Linear attention adds all of these maps into one state:

\[S_t = \sum_{i\le t}v_ik_i^\top, \qquad S_tq = \sum_{i\le t}v_i(k_i^\top q).\]

This gives $S_t$ a concrete interpretation. It is a table of key-to-value associations represented as one linear map. Its fixed shape comes from overlaying every retrieval rule in the same matrix instead of retaining the key-value pairs as separate records. That overlay is also where the problem appears.

Suppose the keys have unit norm and the state is read using one stored key $k_j$:

\[S_tk_j = v_j + \sum_{\substack{i\le t \\ i\ne j}}v_i(k_i^\top k_j).\]

The first term is the value associated with $k_j$. Every other value leaks into the read according to the overlap $k_i^\top k_j$. Orthogonal keys contribute zero, similar keys interfere, and an identical key contributes with weight one. Reusing the same key therefore adds the new value on top of the old one.

This exposes the problem with the additive recurrence. Suppose the current state returns $\widehat v_t=S_{t-1}k_t$ at the address where the model wants to store $v_t$. A plain additive write makes that address return $\widehat v_t+v_t$. For one scalar component, if the old read is $10$ and the new target is $3$, addition produces $13$. Reaching the target requires a correction of $3-10=-7$.

DeltaNet replaces the additive write with exactly this kind of correction. It first reads what is currently stored at the write address, then writes the remaining error (Schlag et al., 2021):

\[\widehat v_t=S_{t-1}k_t, \qquad S_t = S_{t-1} + \beta_t(v_t-\widehat v_t)k_t^\top.\]

In the original DeltaNet, $\beta_t$ is an input-dependent write gate generated as $\beta_t=\sigma(W_\beta x_t)$. The projection $W_\beta$ is a persistent model parameter, while $\beta_t$ is the activation it produces for the current token. Multi-head implementations can generate a separate write strength for each head.

Here $k_t$ is being used as the address for the state update. The model still produces its token output with the separately generated query $q_t$. For a unit-norm key, reading the updated state at the same address gives

\[S_tk_t = (1-\beta_t)\widehat v_t+\beta_tv_t.\]

The difference $v_t-\widehat v_t$ is precisely the correction needed to move the current read toward the target. Multiplying it by $k_t^\top$ writes that correction along the same address direction. When the state is read back with a unit-norm $k_t$, the factor $k_t^\top k_t$ equals one, producing the interpolation above. In the limiting case $\beta_t=1$, the updated map returns exactly $v_t$ at $k_t$. This delta rule is also one online gradient step on the local memory objective

\[\mathcal L_t(S) = \frac{1}{2}\lVert Sk_t-v_t\rVert_2^2.\]

The pretrained projections generate $k_t$, $v_t$, and $\beta_t$, while the recurrence updates the temporary state as the current sequence is read. In this sense, $S_t$ behaves like a small linear map that learns associations from the context.

Gated DeltaNet. A delta update corrects the map around the current address $k_t$, while stale content in other directions can remain. Gated DeltaNet adds a token-dependent decay before the targeted correction, giving the layer a way to weaken the entire old state (Yang et al., 2025):

\[\widetilde S_t=\alpha_tS_{t-1}, \qquad \widehat v_t=\widetilde S_tk_t,\] \[S_t = \widetilde S_t + \beta_t(v_t-\widehat v_t)k_t^\top, \qquad o_t=S_tq_t.\]

The decay gate $\alpha_t\in(0,1)$ controls how much of the existing map survives. A value near one preserves the state, while a value near zero rapidly forgets it. The write gate $\beta_t$ then controls how strongly the current association moves toward $v_t$. The two gates have different jobs: $\alpha_t$ manages the whole memory, and $\beta_t$ edits the direction selected by $k_t$.

That recurrence explains the long-lived state. A complete Gated DeltaNet mixer must also turn the incoming hidden states into the Q, K, and V vectors that read and write it. Write the input to one layer as

\[H= \begin{bmatrix} h_1^\top \\\\ \vdots \\\\ h_T^\top \end{bmatrix} \in\mathbb R^{T\times d_{\text{model}}}.\]

Here $h_t$ is the representation at position $t$. At the first sequence-mixing layer it is still local to the current token; in deeper layers it may already contain context gathered below. The mixer first projects each row into separate Q, K, and V preactivations:

\[U^q=HW_q, \qquad U^k=HW_k, \qquad U^v=HW_v.\]

Without any temporal operation, $U_t^q$ depends only on $h_t$. A short convolution gives every projected channel a small causal window before the recurrent rule sees it:

\[\widetilde U^q_{t,c} = \sum_{r=0}^{K-1}C^q_{r,c}U^q_{t-r,c}.\]

The index $c$ selects one feature channel, and $K$ is a small fixed kernel width. The convolution is depthwise, so each channel has its own $K$ learned temporal weights and the convolution itself does not mix channels. With $K=4$, the Q path therefore has $4d_q$ convolution weights; K and V have separate kernels with $4d_k$ and $4d_v$ weights. Cross-channel mixing has already happened in the preceding linear projections. The convolution is causal, so only positions $t,t-1,\ldots,t-K+1$ contribute. Its shape is unchanged:

\[T\times d_q \longrightarrow T\times d_q.\]

For a width-three example,

\[\widetilde U^q_t = C^q_0\odot U^q_t + C^q_1\odot U^q_{t-1} + C^q_2\odot U^q_{t-2}.\]

One channel can learn a one-token shift, another a local difference, and others combinations of short phrases or boundaries. Separate filters on the K and V channels create a particularly useful possibility in the first layer. For adjacent tokens $A,B$, a key channel at position $t$ can emphasize the projected representation of $A$, while a value channel at the same position emphasizes $B$. The state can then write an association resembling $A\rightarrow B$ within that layer. Without this local path, an earlier layer may first need to move information about $A$ into the representation at $B$. This is a useful mental model for the local induction bias supplied by the convolution; learned filters can take many other forms.

The Gated DeltaNet block described in the paper uses a width-four short convolution. Its Q and K paths each apply a linear projection, ShortConv, SiLU, and $\ell_2$ normalization. The V path applies a linear projection, ShortConv, and SiLU. Figure 10 expands that convolution across four separate tokens and all feature channels, then places it beside the complete mixer.

Read the right half of the figure from $h_t$ upward. Separate learned projections produce the two gates that enter the recurrent rule. The $\alpha_t$ path controls how much of the previous state survives, while the $\beta_t$ path controls the strength of the targeted delta correction. These are the same $\alpha_t$ and $\beta_t$ that appear in the state equations above. A third projection produces $g_t$. Its line bypasses the recurrent-rule box and meets the normalized readout at the multiplication node, showing that this gate acts on the block output without changing $S_t$:

\[y_t = W_O\!\left[ \operatorname{Norm}(o_t) \odot \operatorname{SiLU}(g_t) \right].\]

During generation, the layer caches the recurrent matrix $S_t$ and a short fixed buffer of recent projected QKV channels required by the convolution. The convolution costs $O(TKd)$ over a sequence, with small fixed $K$, and this buffer does not grow with the context. The short convolution handles explicit local mixing; the recurrent matrix carries the longer-lived history. The original 2021 DeltaNet recurrence contains no short convolution. This local path belongs to the fuller Gated DeltaNet block developed later.

Qwen3.5 uses this mixer inside a hybrid stack, repeating three Gated DeltaNet layers followed by one gated global GQA layer. The recurrent layers carry most of the sequence through fixed-size state, while the periodic GQA layers retain token-level KV caches and apply QK-Norm with partial RoPE, restoring direct, query-dependent access to individual past tokens. Qwen reports that this 3:1 mixture performs better than stacks built entirely from linear attention or standard attention, combining efficient recurrent memory with periodic exact retrieval (Qwen Team, 2025).

The mechanisms in this chapter act on distinct parts of the attention path. MQA, GQA, and MLA reduce what is stored for each token. Sliding windows, DSA, and IndexShare change which stored positions a query reads or how those positions are selected. CSA and HCA replace groups of tokens with a shorter sequence of remote-memory entries. Linear attention and DeltaNet replace that growing entry sequence with recurrent state in the layers that use them. QK-Norm, QK-Clip, attention sinks, and output gates control score scale, normalization behavior, or how much of the retrieved result enters the residual stream. Modern models can combine these techniques because stored width, entry count, readable positions, state representation, and attention dynamics are separate design choices.

MLP and MoE

Attention decides which information a position should gather from the sequence. The MLP decides what to compute from the information now stored in that position’s hidden state. It processes each position independently, using the same parameters for every token, and supplies most of the block’s nonlinear transformation.

For $h\in\mathbb R^d$, the original Transformer uses an expanding projection, an activation, and a contracting projection. Omitting biases,

\[\operatorname{FFN}(h) = W_{\mathrm{down}}\, \sigma\!\left(W_{\mathrm{up}}h\right).\]

The intermediate width $d_{\mathrm{ff}}$ is usually larger than the residual width $d$. To see what that extra width provides, let $w_j$ be row $j$ of $W_{\mathrm{up}}$ and let $d_j$ be column $j$ of $W_{\mathrm{down}}$. The same computation can be written as

\[\operatorname{FFN}(h) = \sum_{j=1}^{d_{\mathrm{ff}}} \sigma\!\left(w_j^\top h\right)d_j.\]

This form gives a useful mental model. Each $w_j$ detects a learned pattern in the current hidden state. Its activated response controls how much of the corresponding update direction $d_j$ is written back into the residual stream. Expanding the hidden state creates a larger bank of these detector-and-update pairs; the activation lets different inputs use different combinations of them. Without that nonlinearity, the two projections would reduce to one linear map.

Modern models keep this position-wise structure while making its feature selection more conditional. That change begins with the gated MLP.

Gated MLPs

SwiGLU gives each intermediate coordinate two learned views of the same hidden state. One measures a candidate feature, and the other controls how relevant that feature is in the current context:

\[\operatorname{SwiGLU}(h) = W_{\mathrm{down}} \left[ \operatorname{SiLU}\!\left(W_{\mathrm{gate}}h\right) \odot W_{\mathrm{up}}h \right] = \sum_{j=1}^{d_{\mathrm{ff}}} \underbrace{\operatorname{SiLU}\!\left(w_{\mathrm{gate},j}^{\top}h\right)}_{\text{gate}} \underbrace{\left(w_{\mathrm{up},j}^{\top}h\right)}_{\text{content}} d_j.\]

The product turns each intermediate coordinate into a context-conditioned response. Since $h$ already contains information gathered by attention, the gate can suppress, reverse, or amplify a candidate feature according to that context. It should not be read as a probability: SiLU can be negative and is unbounded above.

This extra projection has a cost, so SwiGLU models commonly use a narrower intermediate dimension than an ordinary two-matrix FFN at a similar parameter budget. The original GLU-variants study found the gated form to improve Transformer quality in controlled comparisons with ReLU- and GELU-based feed-forward blocks (Shazeer, 2020). All six models in this post use this basic SiLU-gated transformation in their dense MLPs, their experts, or both.

DeepSeek V4 adds SwiGLU clamping for training stability. With $u=W_{\mathrm{up}}h$ and $g=W_{\mathrm{gate}}h$, it applies

\[\widetilde u=\operatorname{clip}(u,-10,10), \qquad \widetilde g=\min(g,10), \qquad m=\operatorname{SiLU}(\widetilde g)\odot\widetilde u.\]

The content branch therefore has a two-sided bound, while only large positive gate pre-activations are capped. This limits extreme products without changing the basic design. A gated MLP conditionally selects intermediate features inside one shared function; MoE extends conditional selection to a collection of complete MLPs.

Sparse Mixture of Experts

A gated MLP can choose different intermediate features for different inputs, yet every token still passes through the same $W_{\mathrm{gate}}$, $W_{\mathrm{up}}$, and $W_{\mathrm{down}}$. After attention, one hidden state may describe a fragment of Python, another a Chinese sentence, and another an arithmetic relation. It is plausible that these states benefit from different nonlinear transformations. Making one shared MLP wider gives it more capacity to represent all of them, while every token must pay for the entire wider function.

Mixture of Experts (MoE) gives the layer several independent MLPs and a learned router. Each expert proposes an update to the residual stream; the router decides how strongly that update should contribute for the current hidden state. Let $x=\operatorname{RMSNorm}(h)$ be the normalized block input. With $E$ experts, the router produces one score per expert,

\[s(x)=W_rx, \qquad p(x)=\operatorname{softmax}(s(x)).\]

A dense MoE evaluates every expert and forms a soft mixture of their proposed updates:

\[\begin{aligned} m_{\mathrm{dense}}(x) &= \sum_{e=1}^{E}p_e(x)\,\operatorname{MLP}_e(x),\\ h'&=h+m_{\mathrm{dense}}(x). \end{aligned}\]

This is the first useful MoE intuition: the layer is a small committee of functions. A code token can place more weight on one subset, while a token in another context can prefer a different mixture. If similar hidden states repeatedly favor the same expert, that expert receives more training on that region of the input distribution and can become increasingly useful there. Specialization can sometimes align with recognizable domains or languages. A recent study across ten MoE language models finds experts with measurable domain preferences (Do et al., 2026). These categories remain an empirical interpretation. The architecture itself only learns a partition of hidden-state space, and an expert’s role may have no concise human label.

Dense routing makes that division of labor possible, although evaluating all $E$ MLPs removes the computational advantage of dividing them. In many inputs, only a few router weights matter much. Sparse MoE keeps the $k$ highest-scoring routes and skips the remaining expert computations. Let

\[\mathcal A(x)=\operatorname{TopK}(s(x),k)\]

be the active expert set. One common choice normalizes the selected scores within that set,

\[\widetilde p_e(x) = \frac{\exp(s_e(x))} {\sum_{j\in\mathcal A(x)}\exp(s_j(x))}, \qquad e\in\mathcal A(x),\]

and the layer evaluates only those experts:

\[\begin{aligned} m_{\mathrm{sparse}}(x) &= \sum_{e\in\mathcal A(x)} \widetilde p_e(x)\,\operatorname{MLP}_e(x),\\ h'&=h+m_{\mathrm{sparse}}(x). \end{aligned}\]

The equations describe a concrete sequence: compute $E$ router logits, turn them into probabilities, select the largest $k$, and renormalize the selected weights. Some implementations keep weights normalized across all experts, while others replace softmax with sigmoid scores; those choices change the numerical weights without changing the central idea of conditional expert execution.

Here, sparsity describes execution. The layer still stores all $E$ expert MLPs, while each token activates only $k$ of them. Compared with a dense MoE containing the same experts, sparse MoE has the same total parameter count and substantially less expert computation. Compared with a dense MLP at a similar per-token compute budget, it can use the saved computation to store many more parameters. This is why MoE models often have very large total parameter counts alongside much smaller active parameter counts (Shazeer et al., 2017).

Figure 11 animates the sequence for one token in a complete pre-norm MLP sublayer. Both sides store four expert MLPs and begin with probabilities $(0.41,0.19,0.32,0.08)$. Dense MoE sends the token through all four paths. Sparse top-2 routing selects MLP 1 and MLP 3, renormalizes their weights to $0.56$ and $0.44$, and dispatches the token through those two paths. Their weighted updates are summed and added back to the original $h_t$ through the residual connection.

The hard selection that saves expert computation also changes the learning signal. Sparse routing contains two mathematically different operations. It first maps continuous router scores to a discrete set of expert indices,

\[s\longmapsto \mathcal A(s)=\operatorname{TopK}(s,k).\]

After the set $\mathcal A$ has been chosen, the active mixture remains differentiable:

\[m(x)=\sum_{e\in\mathcal A}\widetilde p_e(x)\operatorname{MLP}_e(x).\]

Gradients pass normally through the selected mixing weights, the selected expert outputs, and the parameters that produced them. The discrete index map is the troublesome piece. Consider Top-1 routing with two scores. If $(s_1,s_2)=(0.60,0.59)$, expert 1 runs. Raising $s_2$ to $0.599$ leaves the selected index and the computation graph unchanged; raising it to $0.601$ switches the active branch to expert 2. The selected index stays constant on either side of the boundary and jumps when the scores cross. Ordinary differentiation cannot assign a useful slope to that jump, so backpropagation treats the chosen set as fixed.

This exposes a second issue: an unselected expert has no counterfactual gradient for that token. Its output was never evaluated, so the loss cannot say whether choosing it would have produced a better update. If normalization occurs before TopK, an unselected router logit may still affect selected probabilities through the shared denominator, yet its expert function remains absent from the forward pass. Repeated routing choices can then reinforce themselves: frequently selected experts see more examples and improve faster, while rarely selected experts receive less training and become even less attractive to the router.

Sparse activation also introduces systems costs. All expert weights still need to be stored, tokens must be dispatched to the selected experts, and distributed training often turns that dispatch into all-to-all communication. A good MoE design therefore has to balance three quantities at once: stored capacity, active computation, and routing balance. Modern models largely share this top-$k$ foundation; their important differences lie in how they arrange shared and routed experts, choose expert granularity, and keep traffic balanced.

Fine-Grained and Shared Experts

Sparse MoE still leaves one important design choice: how wide should each expert be? Modern models often use a large pool of relatively narrow experts. DeepSeekMoE calls this fine-grained expert segmentation (Dai et al., 2024).

The intuition is simple. Suppose one design activates a single expert with intermediate width 8,192, while another activates eight experts of width 1,024. Both evaluate the same total intermediate width for each token. The second design lets the router choose eight smaller transformations independently, giving it many more ways to assemble the update. This is the practical meaning of finer expert granularity: more selectable pieces at a similar active MLP budget.

Some transformations may be useful for almost every token. Leaving all of them inside the routed pool can make several routed experts relearn similar features. In the usual design, a shared expert supplies an MLP path that is evaluated for every token, while the routed experts provide more conditional capacity. Shared experts generally sit outside the routed TopK competition, so they usually do not consume one of the token’s routed slots.

DeepSeek-style MoE implements this in a particularly simple way. The selected routed outputs are mixed first, then one shared SwiGLU output is added directly. If a configuration asks for $S$ shared experts of width $d_{\mathrm{ff}}$, the implementation can represent them as one shared SwiGLU of width $S d_{\mathrm{ff}}$. This works because several always-active SwiGLU branches whose outputs are simply added can be concatenated into one wider branch with exactly the same computation. Under these conditions, one wide shared expert and several narrower shared experts are two parameterizations of the same function. The number mainly describes how much always-active MLP width the layer contains.

These choices appear in slightly different combinations across current models. DeepSeek V4-Pro selects six out of 384 routed experts of width 3,072 and directly adds one shared expert. Kimi K2.5 selects eight out of 384 routed experts of width 2,048 and uses the same direct-add pattern. Qwen3.5 selects ten out of 512 routed experts of width 1,024 and gives its shared path a separate sigmoid gate. GLM-5.2 also directly adds its shared path, while Inkling gives two shared experts input-dependent weights and normalizes them together with the selected routed experts. Finer granularity creates more possible expert combinations; it also brings more router logits, fragmented dispatch, and smaller matrix multiplications.

Keeping the Router Balanced

Specialization does not require every expert to receive exactly the same number of tokens. Large and persistent imbalance is still dangerous. The feedback loop above threatens training, and there is also a systems reason: if one expert receives 70 tokens from a batch while three others receive 10 each, the hardware running the popular expert becomes the bottleneck and the other devices sit partly idle.

The classic solution adds a small balancing objective to the language-model loss. In the Top-1 setting used by Switch Transformer, let $f_i$ be the fraction of tokens actually sent to expert $i$, and let $P_i$ be the average router probability assigned to it. The auxiliary loss is

\[\mathcal L_{\mathrm{balance}} = \alpha E\sum_{i=1}^{E}f_iP_i.\]

When one expert receives both a large token fraction and a large average probability, its term grows. Gradients cannot pass through the discrete assignments inside $f_i$, although they can pass through $P_i$. The router therefore learns to lower persistent preferences for overloaded experts. The coefficient $\alpha$ controls the compromise: too little pressure allows routing collapse, while too much pressure can interfere with the language-model objective and discourage useful specialization (Fedus et al., 2022).

Loss-Free Balancing takes a more direct approach. Let $s_{i,t}$ be the router score of expert $i$ for token $t$. Before TopK selection, the method adds an expert-specific correction bias $b_i$:

\[\mathcal A(x_t) = \operatorname{TopK}_i\!\left(s_{i,t}+b_i,k\right).\]

After each batch, the method counts how many tokens reached each expert. If $c_i$ is expert $i$’s load and $\bar c$ is the average load, it updates

\[b_i \leftarrow b_i+u\,\operatorname{sign}(\bar c-c_i).\]

An underused expert receives a small positive adjustment for the next batch, while an overloaded expert receives a negative one. The bias affects which experts enter TopK; the original router scores still determine how the selected expert outputs are mixed. This makes the correction behave like a feedback controller around the router, with no balancing term added to the language-model gradient (Wang et al., 2024). The same basic correction-bias pattern appears in DeepSeek V4-Pro, Kimi K2.5, GLM-5.2, and Inkling.

The update rate $u$ sets how quickly the controller reacts. A very small value corrects imbalance slowly, while a large value can make loads oscillate. Even with balanced routing scores, the systems layer still has to dispatch tokens efficiently. Capacity-limited implementations may reroute or drop overflow tokens; dropless implementations process them all and wait for the busiest expert. In both cases, keeping the aggregate traffic reasonably balanced is what turns sparse activation into an actual speedup.

Compressing Expert Communication with LatentMoE

In the equations above, each expert looks like another local MLP. At large scale, however, experts are commonly sharded across GPUs through expert parallelism. The important systems object is then the activation that crosses the GPU boundary. After the router selects $k$ experts, a standard MoE dispatches one copy of the current hidden state $h_t\in\mathbb R^d$ to every GPU that owns a selected expert. Each GPU subsequently sends a width-$d$ expert output back for combination. The outgoing and returning all-to-all operations therefore move roughly $kd$ numbers per token in each direction.

The central idea of LatentMoE is to make this payload smaller in both directions. On the originating GPU, one shared down-projection first compresses the hidden state,

\[z_t=W_{\downarrow}h_t, \qquad z_t\in\mathbb R^{\ell}, \qquad \ell<d.\]

The dispatcher now sends $z_t$ rather than $h_t$. Each selected expert consumes and produces an $\ell$-dimensional activation, so the returning expert outputs are also narrow. They are combined while still in the latent space, and only then does one shared up-projection restore the model width. The two data paths are:

\[\text{standard:} \quad h_t\in\mathbb R^d \xrightarrow{\text{dispatch}} E_i \xrightarrow{\text{return}} o_i\in\mathbb R^d,\] \[\text{LatentMoE:} \quad h_t\in\mathbb R^d \xrightarrow{W_{\downarrow}} z_t\in\mathbb R^{\ell} \xrightarrow{\text{dispatch}} E_i \xrightarrow{\text{return}} \widetilde o_i\in\mathbb R^{\ell} \xrightarrow{\text{combine},\,W_{\uparrow}} m_t\in\mathbb R^d.\]

With the same TopK, this changes the activation traffic of each all-to-all from $kd$ to $k\ell$. It can therefore reduce runtime when expert communication is a bottleneck. The down- and up-projections are shared across experts and are each evaluated once per token.

Figure 12 shows how the paper spends this saving on additional conditional capacity. In its illustrative $\ell=d/2$ comparison, the expert pool grows from four to eight and TopK grows from two to four. Each selected expert communicates half as much data, so doubling TopK brings the total per-token all-to-all traffic back to roughly the standard MoE budget. Keeping TopK fixed would instead retain the communication reduction.

The residual path and shared expert remain in the model dimension. The router also continues to read the original $h_t$, preserving the full representation when it decides which experts are relevant. Only the routed expert branch enters the latent space. Its complete output is

\[m(h_t) = W_{\uparrow} \left[ \sum_{i\in\mathcal A(h_t)} p_i(h_t)E_i(W_{\downarrow}h_t) \right].\]

The same compression also narrows the input and output matrices inside every routed expert. If $\ell=d/\alpha$, those expert parameters and operations shrink by roughly a factor of $\alpha$. This is why the iso-budget design can increase both the number of stored experts and the number activated for each token.

MAI-Base-1 provides a concrete example. Its router makes a Top-8 choice among 512 experts from the original 6,656-dimensional hidden state. The selected path compresses that state to 3,072 dimensions before dispatch, and each expert expands the latent representation to an intermediate width of 10,240 before returning a 3,072-dimensional result. After the selected results are combined, the shared up-projection returns them to width 6,656. MAI alternates these LatentMoE layers with ordinary dense MLP layers.

Compression introduces its own tradeoff. A latent space that is too narrow can discard information the experts need, and the additional projections add compute outside the experts. Narrower expert matrices can also become less efficient GPU operations. The saved communication and parameter bandwidth can, however, be reinvested in more experts or a larger TopK, giving the router a richer set of expert combinations at a similar systems budget.

Normalization and Residual Connections

At depth $l$, a decoder represents all $T$ token positions with a matrix

\[H_l = \begin{bmatrix} (h_l^{(1)})^\top \\ \vdots \\ (h_l^{(T)})^\top \end{bmatrix} \in\mathbb R^{T\times d},\]

where row $h_l^{(t)}$ is the current representation of token position $t$. $H_0$ is formed from the input embeddings, and $H_L$ is eventually mapped to vocabulary logits. The sequence

\[H_0\longrightarrow H_1\longrightarrow\cdots\longrightarrow H_L\]

is called the residual stream. The word stream refers to this model-width representation being carried through the depth of the network. It is the $H$ already present in an ordinary Transformer, with no additional module or hidden data structure. Attention reads the rows of $H_l$ and writes an update that can mix information across token positions. The MLP reads each row and writes another update at that position. In generic form, a block performs

\[U_l=H_l+\Delta_l^{\mathrm{attn}}, \qquad H_{l+1}=U_l+\Delta_l^{\mathrm{mlp}}.\]

The existing representation remains on the main path, while each sublayer supplies a correction. If a branch initially produces an update close to zero, the input passes through almost unchanged. A layer can therefore preserve features that are already useful and concentrate its learned capacity on what should be added.

Repeated addition creates a numerical question. Nothing in $H_{l+1}=H_l+\Delta_l$ forces the magnitude of $H_l$ to remain constant as depth and training change. Without normalization, a sublayer would need to interpret both the pattern encoded by a hidden vector and its changing overall scale. These are entangled in the computation. For example, with bias-free linear projections, scaling a hidden state by $c$ also scales its query and key by $c$, so their dot product scales by $c^2$. A previously moderate attention distribution can become much sharper. The inputs to an MLP scale as well, changing its activations and, in a gated MLP, the product between its gate and content branches.

Normalization gives every learned branch a more predictable numerical input. It removes most of the branch’s sensitivity to the overall magnitude of the current residual state, while retaining the coordinate pattern that carries information. This is the central reason normalization appears throughout a Transformer. Its placement then determines whether the residual stream itself is also transformed at every layer.

Pre-Norm and RMSNorm

There are two separate design choices: where normalization is placed, and which normalization function is used. The historical block in the recap uses post-norm. Writing either attention or the MLP as a generic sublayer $F_l$, one post-norm update is

\[H_{l+1} = \operatorname{Norm}\!\left(H_l+F_l(H_l)\right).\]

The sublayer first reads the current hidden state, its result is added to the residual path, and normalization is applied to the sum. The route from $H_l$ to $H_{l+1}$ therefore passes through normalization even when $F_l$ contributes very little.

Most modern decoder-only models instead use pre-norm:

\[H_{l+1} = H_l+F_l\!\left(\operatorname{Norm}(H_l)\right).\]

Here the sublayer reads a normalized copy of $H_l$, while the original $H_l$ travels directly to the addition. For a complete attention-plus-MLP block, the usual form is

\[\begin{aligned} U_l &=H_l+\operatorname{Attention}\!\left(\operatorname{Norm}(H_l)\right), \\ H_{l+1} &=U_l+\operatorname{MLP}\!\left(\operatorname{Norm}(U_l)\right). \end{aligned}\]

This direct copy explains the identity term. In the forward pass, if $F_l$ is zero, then $H_{l+1}=H_l$. The layer does not need to reconstruct its entire input through a newly learned transformation. In the backward pass, the Jacobian of the same update is

\[\frac{\partial H_{l+1}}{\partial H_l} = I+ \frac{\partial F_l(\operatorname{Norm}(H_l))}{\partial H_l}.\]

The $I$ comes from copying $H_l$ directly into the sum: a tensor’s derivative with respect to itself is the identity map. It contributes a correspondingly direct component to the gradient sent toward earlier layers. The learned branch can still amplify, shrink, or redirect the total gradient, so the identity term is not a guarantee of perfect optimization. It does reduce how completely information and gradients depend on repeatedly traversing learned transformations. In post-norm, the normalization Jacobian also acts on the residual route at every layer. Analyses of Transformer initialization found better-behaved gradients for pre-norm blocks and showed that they can be trained with less dependence on learning-rate warm-up (Xiong et al., 2020).

Pre-norm specifies where normalization happens. The remaining choice is how to normalize each token vector $x\in\mathbb R^d$. LayerNorm computes its mean and variance (Ba et al., 2016):

\[\mu(x)=\frac{1}{d}\sum_{i=1}^{d}x_i, \qquad \sigma^2(x)=\frac{1}{d}\sum_{i=1}^{d}\left(x_i-\mu(x)\right)^2,\] \[\operatorname{LayerNorm}(x) = g\odot \frac{x-\mu(x)}{\sqrt{\sigma^2(x)+\epsilon}} +b.\]

Subtracting the mean recenters the coordinates around zero, while division by their standard deviation controls their overall scale. The learned vectors $g$ and $b$ can then rescale and shift individual coordinates.

RMSNorm keeps the scale control and removes the recentering step (Zhang & Sennrich, 2019). It first measures the root mean square of the coordinates,

\[\operatorname{RMS}(x) = \sqrt{\frac{1}{d}\sum_{i=1}^{d}x_i^2+\epsilon},\]

followed by

\[\operatorname{RMSNorm}(x) = g\odot\frac{x}{\operatorname{RMS}(x)}.\]

Ignoring the small $\epsilon$, for any positive scalar $c$,

\[\operatorname{RMSNorm}(cx) \approx \operatorname{RMSNorm}(x).\]

The two vectors have the same coordinate pattern and differ only in overall magnitude, so RMSNorm presents nearly the same input to the branch. This directly addresses the scale problem created by a deep additive stream. It does not force the coordinates to have zero mean. The original RMSNorm work hypothesized that this recentering property was dispensable in many networks and found comparable performance to LayerNorm across its experiments, with a simpler computation.

That empirical tradeoff explains its popularity in large decoder-only models. RMSNorm targets the scale sensitivity that matters to attention and MLP branches, needs only a sum of squares, and omits the mean, centering subtraction, and learned bias used by LayerNorm. Every anchor model in the opening matrix adopts it. This is an effective engineering default supported by empirical results; it does not establish universal superiority over LayerNorm.

One final normalization is still needed because pre-norm only normalizes the copy read by each branch. The main stream continues to accumulate unnormalized additions, so $H_L$ itself can have a depth-dependent magnitude. Before the output head, modern decoders commonly compute

\[Z = \operatorname{RMSNorm}(H_L)W_{\mathrm{vocab}}.\]

This gives the vocabulary projection a controlled input scale, just as the earlier pre-norm operations do for attention and the MLP. Per-layer normalization stabilizes how learned branches read the residual stream; final normalization stabilizes how the output head reads the completed stream.

Manifold-Constrained Hyper-Connections

Pre-norm and RMSNorm make a residual branch easier to optimize, but they leave the connection between layers unchanged. An ordinary residual sublayer always performs

\[x_{l+1}=x_l+F_l(x_l).\]

This fixed identity path is extremely useful: an existing representation can cross the layer directly, and the learned branch only has to supply an update. The limitation lies in the topology. Every layer reads one evolving hidden state and writes its update back into that same state. Features that should survive for many layers and features that need continual processing can still occupy different coordinates, but the residual connection gives them only one explicit cross-layer route. It cannot choose one carried state for the current computation, preserve another through a separate path, or write the new result into several independently routed states.

Hyper-Connections (HC) make that cross-layer route learnable (Zhu et al., 2025). They replace the single residual stream with several full-width streams. A layer reads a learned mixture of them, runs its usual computation, and writes the result back with another learned pattern. Meanwhile, the existing streams can mix and move forward independently. The model now learns both what a layer computes and how information travels between layers.

Suppressing batch and token axes, an ordinary Transformer carries one vector $x_l\in\mathbb R^d$. HC carries

\[X_l = \begin{bmatrix} (x_l^{(1)})^\top \\ \vdots \\ (x_l^{(n)})^\top \end{bmatrix} \in\mathbb R^{n\times d}.\]

The architecture adds the $n$ axis. Each stream still holds a complete $d$-dimensional hidden state. It is unrelated to attention heads, and no part of $d$ is divided among the streams. With batch and sequence dimensions restored, the state changes from $[B,T,d]$ to $[B,T,n,d]$. DeepSeek V4-Pro uses $n=4$ and initializes the four streams by copying the token embedding. Learned routing lets them diverge after that shared starting point.

Why might this help? Multiple streams give the network several explicit paths through depth. Training may learn to carry some features through many layers with small changes, repeatedly update others, and combine selected streams only when a particular sublayer needs them. Their roles are learned rather than assigned by the architecture. Across many layers, the resulting routes can express shorter or longer paths, serial processing, and partially parallel state evolution.

One HC site surrounds an ordinary attention or MLP sublayer $F_l$ with three small mappings:

\[\begin{aligned} h_l^{\mathrm{in}} &= A_lX_l, \\ h_l^{\mathrm{out}} &= F_l(h_l^{\mathrm{in}}), \\ X_{l+1} &= B_lX_l+C_lh_l^{\mathrm{out}}, \end{aligned}\]

where

\[A_l\in\mathbb R^{1\times n}, \qquad B_l\in\mathbb R^{n\times n}, \qquad C_l\in\mathbb R^{n\times1}.\]

$A_l$ is the reader: it combines the $n$ streams into the single input consumed by $F_l$. $B_l$ is the residual mixer: it routes the existing streams into their next-layer versions. $C_l$ is the writer: it distributes the single sublayer output back across the streams. For $n=4$, the complete shape flow is

\[[B,T,4,d] \xrightarrow{\ A_l\; (1\times4)\ } [B,T,d] \xrightarrow{\ F_l\ } [B,T,d] \xrightarrow{\ C_l\; (4\times1)\ } [B,T,4,d],\]

while $B_l\;(4\times4)$ maps the original $[B,T,4,d]$ state directly to another state of the same shape. Figure 13 shows these two paths and the constraint introduced by mHC.

The paper calls these mappings $H_l^{\mathrm{pre}}$, $H_l^{\mathrm{res}}$, and $H_l^{\mathrm{post}}$. The notation here uses $A_l=H_l^{\mathrm{pre}}$, $B_l=H_l^{\mathrm{res}}$, and $C_l=(H_l^{\mathrm{post}})^\top$ to keep the read, mix, and write roles compact. All three mappings sit outside $F_l$ and are unrelated to the Q, K, and V projections inside attention. Their dimensions are fixed by $n$, while their values can depend on the current token and layer through dynamic projections of the multi-stream state.

HC adds cross-layer capacity without executing the expensive sublayer $n$ times. $A_l$ first reduces the stream axis, so attention or the MLP still receives $[B,T,d]$ and runs once at its ordinary width. The persistent residual tensor does contain $n$ times as many elements. Extra costs therefore include residual activation storage, memory traffic, and the small routing operations. QKV, attention, and MLP activations retain their usual widths, so the full block’s activation memory and FLOPs do not scale uniformly by $n$.

The freedom of HC replaces the fixed identity route with a learned mixer. In an ordinary residual connection, repeatedly composing the skip path still gives $I$. In HC, an early state reaches a later layer through products such as

\[B_{L-1}B_{L-2}\cdots B_lX_l.\]

A free matrix can amplify some combinations of streams and suppress others. Even a recurring gain of $1.02$ becomes $1.02^{100}\approx7.2$ after 100 sites, while $0.98^{100}\approx0.13$. With full matrices, the sensitive directions can also rotate from layer to layer. The mHC paper reports composite forward or backward gains peaking around 3,000 in an unstable 27B HC run. HC solves the rigidity of a fixed residual topology, then exposes a new optimization problem: unconstrained routing matrices are repeatedly multiplied through depth.

Manifold-Constrained Hyper-Connections (mHC) preserve HC’s multi-stream topology and restrict the routing so that this deep composition stays controlled (Xie et al., 2026). The residual mixer $B_l$ is projected into the Birkhoff polytope, the set of non-negative doubly stochastic matrices:

\[\mathcal B_n = \left\{ B\in\mathbb R^{n\times n} \;\middle|\; B\mathbf1=\mathbf1, \ \mathbf1^\top B=\mathbf1^\top, \ B_{ij}\ge0 \right\}.\]

Each row summing to one makes an output stream a convex mixture of the inputs. Each column summing to one preserves the total influence of an input across the outputs. If all streams carry the same vector $x$, then $B_lX_l=X_l$ because every row combines identical vectors with weights that sum to one. This gives the multi-stream system an identity-like common mode. The column condition preserves the mean across streams. A doubly stochastic matrix also has spectral norm at most one, so the residual mixer cannot amplify the Euclidean norm. It may contract differences between streams, and it does not preserve every direction exactly.

These properties survive depth because a product of doubly stochastic matrices remains doubly stochastic. Geometrically, every such matrix is a convex combination of permutation matrices. The mixer can softly rearrange and blend streams while keeping the same global constraints after many layers. This directly targets the matrix-product instability introduced by HC.

The model begins with an unconstrained $\widetilde B_l$, makes its entries positive, and alternates column and row normalization through Sinkhorn-Knopp iterations:

\[M^{(0)}=\exp(\widetilde B_l), \qquad M^{(r)} = \operatorname{RowNorm} \left( \operatorname{ColNorm}(M^{(r-1)}) \right).\]

Column normalization balances how much total influence each source stream contributes. Row normalization makes every destination a convex mixture. Repeating both operations approaches the two conditions simultaneously. DeepSeek uses 20 iterations and sets $B_l=M^{(20)}$. The reader and writer receive simpler bounds:

\[A_l=\sigma(\widetilde A_l), \qquad C_l=2\sigma(\widetilde C_l).\]

Their coefficients remain non-negative and bounded, reducing fragile cancellation between large positive and negative routes. The underlying logits are still generated from the current normalized multi-stream state together with learned static terms, so the constrained routing remains token-dependent.

DeepSeek V4-Pro places one mHC site around attention and another around the MoE in every block. Its residual state has shape $[B,T,4,7168]$. At each site, $A_l$ reduces four streams to one 7,168-dimensional $h_l^{\mathrm{in}}$, the pre-RMSNorm sublayer runs once, and $C_l$ writes $h_l^{\mathrm{out}}$ into the streams already mixed by $B_l$. After 61 blocks, a learned hyper-connection head reduces the four streams before the final RMSNorm and vocabulary head.

The four-stream residual state still increases memory traffic, and dynamic mapping generation plus Sinkhorn adds work. DeepSeek reports that fused kernels, selective recomputation, and pipeline overlap reduce the measured training wall-time overhead to 6.7% for its optimized four-stream implementation. The main architectural economy comes from executing each attention and MoE sublayer once at width $d$.

If the matrices still feel abstract, picture the residual path as a road system. An ordinary residual connection is a single lane: every layer preserves the existing traffic and adds its new output onto the same road. HC opens four lanes and adds learned ramps. $A_l$ chooses which lanes feed the current sublayer, $C_l$ decides where its new output enters, and $B_l$ controls how existing traffic changes lanes. This makes routing more flexible, but free ramps can repeatedly concentrate or dilute particular traffic patterns. mHC places a balanced-traffic rule on $B_l$: old traffic may be redistributed among lanes without acquiring an exponentially growing gain from the routing itself. New information can still enter through $C_l$.

The Design Logic of Modern Transformers

A modern Transformer is best understood as a hierarchy of information paths with different costs, fidelity, and access patterns. Exact token-level memory and dense computation are expensive, and modern architectures increasingly treat them as separate budgets. A model may preserve one while economizing another through compressed, sparse, recurrent, or conditional paths. The memory, bandwidth, computation, and communication saved along those paths can support greater parameter capacity and more flexible routing elsewhere. Precise paths and numerical constraints then protect the capabilities placed most at risk by that approximation.

Seen through this lens, the changes to different Transformer components belong to one system. Attention has become memory design: it determines what history is retained, in what representation, and which parts remain directly addressable. Positional mechanisms attach location in a form compatible with that representation. The MLP has become conditional computation: MoE stores a large collection of transformations while activating only a small subset for each token. Residual connections determine how information travels across depth, and normalization, gates, and routing constraints keep those paths controlled. The familiar decoder block remains the interface; the resource allocation inside it has changed.

The attention mechanisms in this post occupy different points in that allocation. MLA keeps one entry per token and reduces the feature width of each entry. Local and sparse attention retain token-level memories while limiting which ones a query reads. CSA and HCA replace parts of the remote sequence with learned summaries. Gated DeltaNet folds the history into a fixed-size recurrent state. These choices reduce different costs and preserve different capabilities. A latent cache keeps token-level addressing. Sparse retrieval preserves access to selected remote tokens. Block compression reduces the number of memories that continue to exist. Recurrent state removes the growing token axis and gives up independent access to the records folded into it.

MoE applies the same allocation principle to model capacity. Sparse routing makes total parameters and per-token computation partially independent, while fine-grained experts give the router more ways to compose an update. LatentMoE extends the calculation to communication by narrowing the activation sent between devices. The resulting savings often become capacity elsewhere. DeepSeek V4, for example, reduces remote-memory and active-expert costs while carrying a much larger parameter pool and four residual streams. Architectural efficiency is therefore best understood as budget reallocation: one resource becomes cheaper so another form of capacity becomes affordable.

The strongest recurring design principle is hybridization. Aggressive approximations often preserve a narrower path with higher fidelity. Qwen periodically restores full attention between recurrent layers, while DeepSeek keeps an exact local window beside compressed remote memory. Local stacks reconnect through global layers, and routed experts may coexist with dense or shared computation. These paths preserve access to information that a compressed or conditional route may lose. Constraints address a separate risk. QK normalization controls attention-score growth, router balancing controls expert traffic, and mHC constrains residual mixing through depth. Exact paths preserve access; constraints preserve control.

The intended allocation must also survive implementation. MLA saves cache only when the runtime retains the latent representation. Local attention saves cache capacity only when unreachable entries are evicted. Sparse attention saves work only when the kernel consumes the selected indices directly. Gated DeltaNet needs a recurrence kernel that exploits its fixed-size state, and LatentMoE must keep activations narrow across the device boundary. At this scale, an architecture specifies where tensors live and how they move almost as much as it specifies the equations they satisfy.

Across these models, recent detail commonly receives an explicit local path, while remote history may remain latent, be selected on demand, become a learned summary, or be folded into recurrent state. Broadly useful transformations remain dense where consistency matters, while additional capacity becomes conditional. The resources most often reduced are KV-cache capacity, memory bandwidth, attention work, active expert computation, and communication traffic. Those savings support larger expert pools, richer routing, additional residual state, and periodic exact access. Modern Transformer design is the construction of a budgeted information hierarchy: preserve exactness where errors are costly, compress or route the remaining work, and spend the savings on capacity that each token can use selectively.