Normalizations

This article studies normalization in the context of a language model: token IDs become embeddings, attention and FFN layers modify hidden states, and an output head produces next-token logits. At each stage, a vector’s direction and magnitude can affect the next computation. Normalization changes which of these differences that computation can use.

The main path is token representations, LN/RMSNorm, attention scaling and QK-Norm, and residual connections. L2 normalization helps explain the geometry. AdaLN shows how conditions can modify normalized features. BN, IN, GN, and other families provide comparisons, including uses in vision components of multimodal systems; their inclusion does not imply that they are standard replacements inside a causal text decoder. Each time, ask: which numbers are grouped together, what information is removed on this path, and where can the complete model still use it?

本文在 language model 的语境下理解 normalization：token IDs 先变成 embeddings，attention 与 FFN layers 不断修改 hidden states，最后由 output head 产生 next-token logits。在每个位置，向量的方向与 magnitude 都可能影响后续计算。Normalization 改变的是后续计算能够利用哪些差异。

全文主线是 token representations、LN/RMSNorm、attention scaling 与 QK-Norm，以及 residual connections。L2 normalization 帮助解释几何关系，AdaLN 展示 condition 如何调节 normalized features；BN、IN、GN 等家族则作为对照，也涉及 multimodal 系统中的 vision components。收录这些方法，不代表它们都是 causal text decoder 中的常规替代品。每遇到一种方法，都问：哪些数被放到一起计算？这条路径去掉了哪些信息？完整模型又能在哪里继续使用这些信息？

At one LM layer, suppose a token could have hidden state $(1,2,3)$ or $(101,102,103)$. These are two hypothetical states at the same computational site. The second has the same differences between coordinates, but a common offset of 100. A third state, $(10,20,30)$, multiplies both the values and their differences by ten. A coordinate is one entry of the vector; a vector with $d$ coordinates has dimension $d$.

These changes matter to the next computation. A fixed linear map sees a tenfold input scale as a tenfold output scale, before any bias. A nonlinear function can react even more differently: large differences between attention scores can push softmax into its flat, nearly one-hot regime. Attention and Language Models illustrates this effect.

Where could these changes come from inside an LM? A token’s vector does not stay at its initial embedding. As it passes through the network, successive computations revise it using context and learned features. In a GPT-style block, attention mixes information from visible token positions, while an FFN transforms each position’s features. Each is a sublayer: one computation within the larger block.

To understand how one sublayer revises the representation, call the incoming token vector $x$ and the sublayer’s computed contribution $u$. After any required output projection, $u$ has the same dimension as $x$, so the two can be added coordinate by coordinate:

\[x_{\mathrm{new}}=x+u.\]

This addition is called residual addition. The original $x$ reaches the addition directly, and the sublayer supplies a learned change $u$. The resulting $x_{\mathrm{new}}$ is passed onward as the token’s updated representation. Here “updated” refers to a hidden state changing during a forward pass; the model parameters need not change. We will place normalization within this computation in Section 6.1.

Now the scale question becomes concrete. If $x=(1,2,3)$ and a sublayer produces $u=(10,10,10)$, the next representation is $(11,12,13)$. If the following sublayer adds the same contribution, it becomes $(21,22,23)$. The common offset has grown even though the differences between coordinates are unchanged.

Magnitude can change as well. If a sublayer happens to contribute a vector equal to its incoming $x$, the addition produces $2x$. Repeating that particular situation would double the state at every addition. These are possible numerical examples, not a claim that trained sublayers always produce such contributions. They establish the limited point we need: residual addition preserves a route for the incoming representation, but the addition itself does not keep its mean or magnitude fixed. Consequently, later computations can receive vectors on different numerical scales.

Normalization provides a reference for a particular computation. This is a modeling choice with an information cost: removing a common scale makes that computation unable to distinguish some inputs. In an LM, we must first understand what scale can do, then examine where normalization occurs.

假设在 LM 的某一层，一个 token 可能具有 hidden state $(1,2,3)$，也可能具有 $(101,102,103)$。我们比较的是同一计算位置的两种假设状态。第二个向量各 coordinate 之间的差值没变，但整体多了 100 的共同偏移。第三种状态 $(10,20,30)$ 则把数值及其差值都放大了十倍。Coordinate 就是向量中的一个数；有 $d$ 个 coordinates，向量维度就是 $d$。

这些变化会影响下一步计算。对于固定的 linear map，输入放大十倍，加入 bias 之前的输出也放大十倍。Nonlinear function 的反应可能更明显：attention scores 的差距变大，softmax 就可能进入平坦、接近 one-hot 的区域。Attention and Language Models 用图解释了这种现象。

这些变化在 LM 内部可能从哪里来？一个 token 的向量不会一直停留在初始 embedding。随着它经过 network，后续计算会利用 context 与 learned features 逐步修改它。在 GPT-style block 中，attention 混合可见 token positions 的信息，FFN 则变换每个位置的 features。它们各自是一个 sublayer，也就是较大的 block 内的一步计算。

先看一个 sublayer 怎样修改表示。把进入它之前的 token vector 记为 $x$，把它计算出的贡献记为 $u$。经过所需的 output projection 后，$u$ 与 $x$ 的维度相同，因此可以逐 coordinate 相加：

\[x_{\mathrm{new}}=x+u.\]

这个加法就叫 residual addition。原始 $x$ 可以直接到达加法的位置，sublayer 则提供一个学到的修改量 $u$；相加得到的 $x_{\mathrm{new}}$ 作为该 token 的新表示，继续传给后续计算。这里的“修改”发生在一次 forward pass 中，修改的是 hidden state，不要求 model parameters 同时发生变化。第 6.1 节再说明 normalization 可以放在这段计算的什么位置。

现在就能具体观察尺度问题。若 $x=(1,2,3)$，某个 sublayer 计算出 $u=(10,10,10)$，下一个表示就是 $(11,12,13)$。如果后面的 sublayer 又加上同样的贡献，就变成 $(21,22,23)$。虽然各 coordinates 之间的差值没变，共同 offset 却持续增大了。

Magnitude 也可能改变。如果某个 sublayer 恰好贡献了与输入 $x$ 相同的向量，相加就得到 $2x$；如果后续每次都出现这种情况，state 就会每次翻倍。这些是可能的数值例子，不是说训练好的 sublayers 总会产生这样的贡献。它们说明的是：residual addition 给原有表示保留了一条传递路径，但这个加法本身并不让 mean 或 magnitude 保持固定。 因此，后续计算可能收到处于不同数值尺度的向量。

Normalization 为特定计算提供参照，但这是一项具有信息代价的建模选择：去掉共同尺度，就会让这一步计算无法区分某些输入。在 LM 中，我们需要先理解尺度能做什么，再讨论 normalization 应该放在哪里。

It can. A norm measures magnitude, and magnitude can change how a representation affects the model. There is no universal rule that a larger hidden-state norm means a more important token or a more confident answer. Its meaning depends on where the vector is used. The following examples hold other quantities fixed so that we can isolate each effect.

Attention scores: how selective is the read? A query $q$ is compared with candidate keys $k_j$ using dot products. Holding the keys fixed, multiplying the query by a positive number multiplies all its dot-product scores by that number, provided no intervening normalization removes the change. Scores $(1,0)$ give softmax weights about $(0.731,0.269)$; scores $(10,0)$ give about $(0.99995,0.00005)$. The preferred key stays the same, but selection becomes much sharper. Query scale can therefore control selectivity. Scaling an individual key changes its own score; if that score is negative, increasing its magnitude makes it more negative rather than more attractive.

Value vectors: how much is contributed? After selecting weights, attention forms a weighted sum of value vectors. If a key receives weight 0.2, its contribution is $0.2v$. Replacing its value by $10v$, while keeping the weights fixed, makes that contribution ten times larger. Other contributions can reinforce or cancel it, and an output projection can further change it. The attention weight alone therefore does not determine the size of the resulting contribution. These examples follow directly from the attention computation.

Residual states: how much can a new update change the representation? A residual stream is the hidden vector carried between sublayers. Each sublayer adds an update $u$, producing $x+u$. For a fixed $u$ that is not parallel to $x$, adding it to $x$ changes the direction more than adding it to $10x$. For example, $(1,0)+(0,1)=(1,1)$ turns by 45 degrees, while $(10,0)+(0,1)=(10,1)$ turns by only about 5.7 degrees. Thus the existing state’s scale helps determine how strongly the next sublayer can alter its direction.

Output logits: how concentrated is the prediction? Logits are the scores immediately before the vocabulary softmax. For an illustrative two-token vocabulary, logits $(1,0)$ versus $(10,0)$ again give probabilities about $(0.731,0.269)$ versus $(0.99995,0.00005)$. The ranking is unchanged, but the assigned likelihoods—and the cross-entropy loss—are very different. A sharper distribution need not be more correct. This example scales the logits themselves; scaling a hidden state before a final LN/RMSNorm does not necessarily scale the logits, because that normalization can remove the change.

These are computational roles that scale can play, not evidence that every trained LM gives norm a particular semantic meaning. Whether a specific model uses it for a particular task requires examining that model.

Why normalize, then? We choose which paths should depend on absolute scale and which should have a stable reference. LN and RMSNorm restrict scale dependence on their normalized path; a Pre-Norm residual path can still preserve scale and let it affect later computation. Section 6.1 works through this distinction. Training may also represent useful distinctions through coordinate patterns that survive normalization; this is a possibility enabled by learning, not a guarantee that discarded information is always recovered.

可能包含。 Norm 测量 magnitude，而 magnitude 可以改变一个表示如何影响模型。但不存在通用规则说“hidden-state norm 越大，token 越重要”或“答案越可信”。它的意义取决于这个向量在哪里参与计算。下面固定其他量，只改变一处尺度，分别观察效果。

Attention scores：读取时有多偏向某个位置？ Query $q$ 通过 dot product 与候选 keys $k_j$ 比较。固定 keys，如果把 query 乘以一个正数，并且中间没有 normalization 抵消这个变化，这一行 dot-product scores 就会一起乘以该数。Scores $(1,0)$ 对应 softmax weights 约 $(0.731,0.269)$；scores $(10,0)$ 则对应约 $(0.99995,0.00005)$。偏好的 key 没变，但选择变得尖锐得多。因此，query scale 可以控制读取的选择性。如果只放大一个 key，则只改变它自己的 score；原 score 为负时，放大 magnitude 会让它更负，并非更受偏好。

Value vectors：实际贡献多大？ 确定 weights 后，attention 对 value vectors 加权求和。假设一个 key 的 weight 为 0.2，其贡献就是 $0.2v$。保持 weights 不变，把该 value 换成 $10v$，这项贡献就放大十倍。其他 contributions 可以与它叠加或抵消，后续 output projection 也会继续变换它。因此，仅看 attention weight，还不能确定最终贡献的大小。这些例子直接来自 attention 的计算过程。

Residual states：新 update 能改变表示多少？ Residual stream 是在 sublayers 之间持续传递的 hidden vector；每个 sublayer 加入 update $u$，得到 $x+u$。固定一个不平行于 $x$ 的 $u$，把它加到 $x$ 上与加到 $10x$ 上，方向变化程度不同。例如 $(1,0)+(0,1)=(1,1)$，方向转过 45 度；$(10,0)+(0,1)=(10,1)$，方向只转过约 5.7 度。因此，原有 state 的尺度会影响下一 sublayer 对其方向的修改强度。

Output logits：预测概率有多集中？ Logits 是 vocabulary softmax 之前的 scores。用只有两个候选 tokens 的简化词表举例，logits $(1,0)$ 与 $(10,0)$ 分别对应约 $(0.731,0.269)$ 与 $(0.99995,0.00005)$ 的概率。排名相同，但赋给目标的 likelihood、以及 cross-entropy loss 都可能非常不同。更尖锐的分布不代表更正确。这里缩放的是 logits 本身；如果缩放的是 final LN/RMSNorm 之前的 hidden state，变化可能被 normalization 抵消，并不一定会放大 logits。

这些是尺度能够发挥的计算作用，不是说每个训练好的 LM 都把 norm 当成某种固定的语义信号。具体模型是否这样利用尺度，需要检查该模型本身。

既然尺度有用，为什么还要 normalize？因为我们在选择：哪些计算路径应该依赖绝对尺度，哪些路径应该使用稳定参照。LN 与 RMSNorm 限制 normalized path 对尺度的依赖；Pre-Norm 的 residual path 仍可保留尺度，并让它影响后续计算。第 6.1 节会具体展开。训练也可能把有用区别编码到 normalization 后仍能保留的 coordinate patterns 中；这是学习提供的一种可能性，不保证被丢掉的信息总能恢复。

First choose a set $S$ containing $m$ numbers. Our starting case is the hidden coordinates of one token at one layer; other statistical groups will be compared in Section 3. Compute their mean and variance:

\[\mu_S=\frac1m\sum_{j\in S}x_j,\qquad \sigma_S^2=\frac1m\sum_{j\in S}(x_j-\mu_S)^2.\]

The $m$ in both denominators counts the entries selected into $S$. For a token vector $(1,2,3)$, $S$ contains its three coordinate indices, so $m=3$. The formulas become $\mu_S=(1+2+3)/3=2$ and $\sigma_S^2=((1-2)^2+(2-2)^2+(3-2)^2)/3=2/3$. In token-wise LayerNorm with hidden width $d$, $m=d$, regardless of batch size or the number of tokens in the dataset.

The mean specifies the common level. The variance $\sigma_S^2$ measures the average squared distance from that level. Its square root, the standard deviation $\sigma_S$, has the same units as the original numbers. Throughout the article, $\sigma^2$ denotes variance and $\sigma$ denotes standard deviation when describing these statistics. Centering and dividing by that scale gives

\[\hat x_i=\frac{x_i-\mu_S}{\sqrt{\sigma_S^2+\epsilon}},\qquad y_i=\gamma_i\hat x_i+\beta_i.\]

Here $\epsilon>0$ prevents division by zero. There are two stages in the formula. First, compute $\mu_S,\sigma_S^2$ from the input and use them to obtain $\hat x_i$. Then multiply by a learned scale $\gamma_i$ and add a learned shift $\beta_i$. These last two numbers let the model adjust what the next layer receives.

Why adjust values after standardizing them? Zero mean and approximately unit variance provide a convenient reference, but they are not necessarily the most useful output statistics for the next computation. Stopping at $\hat x$ would require every input group to keep that reference. The affine stage lets training choose how to use it: $\gamma_i$ adjusts how strongly coordinate $i$ varies, while $\beta_i$ changes its baseline.

For example, suppose a standardized coordinate takes values $-1,0,1$ across three inputs. Setting $\gamma_i=2,\beta_i=3$ maps them to $1,3,5$. The relative variation remains, but its size doubles and the baseline moves to 3. Those particular values are just an illustration; training learns the parameters from the loss. Both changes can affect which region of a subsequent nonlinear function the feature reaches.

This second stage is optional. Setting every $\gamma_i=1$ and $\beta_i=0$ gives the standardized values unchanged, and a layer can omit these parameters entirely. If a freely learned linear map follows immediately, the affine transform can also be absorbed into that map’s weights and, where available, bias. Its usefulness therefore depends on the surrounding architecture; normalization does not mathematically require trainable parameters.

To see where these numbers come from, consider two tokens entering the same normalization layer, each represented by three coordinates:

\[x_A=(1,2,3),\qquad x_B=(10,20,30).\]

For this example, choose the token-wise rule that we will later call LayerNorm. Token A computes its mean from its own three numbers: $\mu_A=2$, with variance $\sigma_A^2=2/3$. Token B separately computes $\mu_B=20$ and $\sigma_B^2=200/3$. When standardizing A, all three coordinates subtract the same 2 and divide by the same $\sqrt{2/3+\epsilon}$. This reuse of one pair of statistics is what “sharing statistics” means. B uses its own pair; the two tokens are not pooled together.

Now consider the second stage. This layer stores three learned scales $(\gamma_1,\gamma_2,\gamma_3)$ and three learned shifts $(\beta_1,\beta_2,\beta_3)$. Unlike the means and variances, these six parameters are not recalculated from each token. They are model parameters, updated during training and reused wherever this layer is applied.

The first coordinate of either token uses $\gamma_1,\beta_1$. The second uses $\gamma_2,\beta_2$, and the third uses $\gamma_3,\beta_3$. “Sharing parameters” means reusing the same stored parameters, rather than giving each token a separate set. For example, if $\gamma_1=2$ and $\beta_1=0$, this layer doubles the standardized first coordinate of both tokens.

Compare A’s first and second coordinates: they share statistics but use different parameter entries. Compare A’s and B’s first coordinates: they use separately computed statistics but share parameters. We therefore need to specify both which inputs are measured together and where each learned scale and shift is reused. Later methods will make different choices.

Ignoring $\epsilon$, both example tokens become approximately $(-1.225,0,1.225)$ before the learned scale and shift. With $\epsilon>0$, each standardized group’s variance is $\sigma_S^2/(\sigma_S^2+\epsilon)$, slightly below 1. After the learned affine transform, neither zero mean nor unit variance is guaranteed.

Does this undo normalization? In the idealized example with $\epsilon=0$, A and B have already become identical. Applying the same learned scales and shifts keeps them identical; it cannot recover A’s original scale of 1 and B’s original scale of 10. Normalization removes the input-dependent common scale, while the affine parameters supply a learned scale reused across inputs. The model gets a choice of output range without automatically restoring the discarded input-specific information.

What exactly is being “normalized”? (Click to expand)

Standardization fixes a group’s first two moments: mean and variance. It does not turn arbitrary data into a Gaussian distribution, remove correlations between coordinates, or guarantee bounded individual values. A single outlier can still be large relative to the group’s average scale.

The $m$ is the group size in the $1/m$ factors of the mean and variance formulas above. For our three-coordinate token, $m=3$; for token-wise LN at hidden width $d$, $m=d$. Dividing by $m$ defines the variance of the actual group being transformed. Some statistical estimators divide by $m-1$ to estimate an underlying population variance without bias from independent samples whose mean is also estimated. Normalizing a token’s coordinates does not require treating them as such a sample. These divisors should not be silently interchanged in an implementation.

先选一组数 $S$，其中有 $m$ 个元素。我们先看 LM 某一层中，一个 token 的 hidden coordinates；第 3 节再比较其他统计分组。计算 mean 与 variance：

\[\mu_S=\frac1m\sum_{j\in S}x_j,\qquad \sigma_S^2=\frac1m\sum_{j\in S}(x_j-\mu_S)^2.\]

两个公式分母中的 $m$，就是选进 $S$ 的数的个数。对于 token vector $(1,2,3)$，$S$ 包含它的三个 coordinate indices，因此 $m=3$。公式展开为 $\mu_S=(1+2+3)/3=2$，以及 $\sigma_S^2=((1-2)^2+(2-2)^2+(3-2)^2)/3=2/3$。逐 token LayerNorm 的 hidden width 为 $d$ 时，$m=d$；它不是 batch size，也不是数据集中的 token 总数。

Mean 描述共同的数值水平。Variance $\sigma_S^2$ 衡量这些数与共同水平之间的平均平方距离；开平方得到 standard deviation $\sigma_S$，它与原始数值具有相同单位。全文描述这些统计量时，统一用 $\sigma^2$ 表示方差、$\sigma$ 表示标准差。先去掉共同水平，再除以这个尺度：

\[\hat x_i=\frac{x_i-\mu_S}{\sqrt{\sigma_S^2+\epsilon}},\qquad y_i=\gamma_i\hat x_i+\beta_i.\]

$\epsilon>0$ 用于防止除零。上面的公式实际上分两步：第一步从输入中算出 $\mu_S,\sigma_S^2$，用它们得到 $\hat x_i$；第二步再乘上可训练的 scale $\gamma_i$，加上可训练的 shift $\beta_i$，让模型调整下一层收到的数值。

为什么已经标准化，还要再调整？因为零均值、接近单位方差提供了方便的参照，但不意味着下一步计算最需要的输出就应该永远具有这个范围。如果停在 $\hat x$，每组输入就必须保持这个参照。加入 affine transform 后，训练可以决定如何使用它：$\gamma_i$ 调整第 $i$ 个 coordinate 的变化幅度，$\beta_i$ 调整它的基准位置。

例如，假设某个 coordinate 标准化后，在三个不同输入中分别取值 $-1,0,1$。令 $\gamma_i=2,\beta_i=3$，它们就变成 $1,3,5$：相对变化仍然保留，变化幅度放大两倍，基准位置移到 3。这些具体数值只是示例，实际参数由训练根据 loss 学习。如果后面接 nonlinear function，幅度与基准的变化还会影响这个 feature 落入函数的哪个区域。

这第二步是可选的。 把所有 $\gamma_i$ 设为 1、$\beta_i$ 设为 0，就原样输出标准化后的结果；也可以完全不设置这些参数。如果后面立即接一个可自由学习的 linear map，这个 affine transform 还可以合并进后续的 weights，以及存在时的 bias。因此它的用途取决于周围架构，normalization 在数学上并不要求可训练参数。

这些数分别从哪里来？我们用一个具体例子走一遍。假设两个 tokens 进入同一层 normalization，每个 token 用三个 coordinates 表示：

\[x_A=(1,2,3),\qquad x_B=(10,20,30).\]

这个例子采用后面会介绍的逐 token LayerNorm 规则。Token A 用自己的三个数计算，得到 mean $\mu_A=2$、variance $\sigma_A^2=2/3$。Token B 单独用自己的三个数计算，得到 $\mu_B=20$、$\sigma_B^2=200/3$。对 A 做标准化时，三个 coordinates 都减去同一个 2，再除以同一个 $\sqrt{2/3+\epsilon}$。这就是“共享统计量”的意思：三个数复用同一对 mean 与 variance。B 使用另一对统计量，不会把两个 tokens 混在一起计算。

再看第二步。这一层保存了三个 learned scales $(\gamma_1,\gamma_2,\gamma_3)$ 和三个 learned shifts $(\beta_1,\beta_2,\beta_3)$。与 mean、variance 不同，这六个参数不是每次从某个 token 的数值中重新算出来的。它们属于模型参数，通过训练更新，并在这一层处理不同 tokens 时反复使用。

具体来说，无论处理 A 还是 B，第一个 coordinate 都使用 $\gamma_1,\beta_1$，第二个使用 $\gamma_2,\beta_2$，第三个使用 $\gamma_3,\beta_3$。“共享参数”就是复用模型中保存的同一组参数，而不是给每个 token 单独准备一套。例如，如果 $\gamma_1=2,\beta_1=0$，这一层就会把 A 和 B 标准化后的第一个 coordinate 都乘以 2。

现在比较 A 的第 1、2 个 coordinates：它们共享统计量，却使用不同的参数项。再比较 A 和 B 的第 1 个 coordinate：它们使用各自计算的统计量，却共享参数。因此，我们需要分别说明哪些输入放到一起测量，以及每个 learned scale 与 shift 会被哪些位置复用。后面的方法会对这两个问题作出不同选择。

暂时忽略 $\epsilon$，上面两个 tokens 在 learned scale 与 shift 之前都会变成约 $(-1.225,0,1.225)$。加入 $\epsilon$ 后，每组标准化结果的 variance 是 $\sigma_S^2/(\sigma_S^2+\epsilon)$，略小于 1。再经过 learned affine transform 后，就不再保证零均值或单位方差。

这样是否又把 normalization 撤销了？在 $\epsilon=0$ 的理想化例子中，A 与 B 已经变得相同；再对它们使用同一组 learned scales 与 shifts，结果仍然相同，无法分别恢复 A 原来的 1 倍尺度和 B 原来的 10 倍尺度。Normalization 去掉的是随输入变化的共同尺度，affine parameters 提供的是在不同输入之间复用的 learned scale。模型因此可以选择输出范围，但不会自动找回已丢掉的、每个输入特有的尺度信息。

“标准化”到底保证了什么？（点击展开）

Standardization 调整的是一组数的 mean 与 variance。它不会自动把数据变成 Gaussian distribution，不会消除 coordinates 之间的相关性，也不保证每个数都被限制在一个固定小区间。单个 outlier 相对于整组数的典型尺度，仍然可能很大。

这里的 $m$ 就是上方 mean、variance 公式中 $1/m$ 的分母，表示本次统计组内的元素个数。我们的三维 token 中 $m=3$；hidden width 为 $d$ 的逐 token LN 中，$m=d$。除以 $m$ 是计算当前这组数本身的 variance。有些统计估计量把独立样本的均值也从样本中估计出来，再除以 $m-1$，以无偏估计背后的总体方差。对一个 token 的 coordinates 做 normalization，并不要求把它们当成这样的独立样本。实现时不能随意互换这两个分母。

A vector’s Euclidean length is $\lVert x\rVert_2=\sqrt{\sum_i x_i^2}$. This norm is a scalar measurement. L2 normalization uses it to transform the vector:

\[u=\frac{x}{\max(\lVert x\rVert_2,\epsilon)}.\]

For $x=(3,4)$, the length is 5 and the result is $(0.6,0.8)$. For $(30,40)$ the result is the same. Except near the numerical floor, the output has length 1 and preserves direction. There is no mean subtraction, so $(3,4)$ and $(103,104)$ generally point in different directions and normalize differently.

Why use this for embeddings? A dot product combines length and alignment:

\[x^\top y=\lVert x\rVert_2\lVert y\rVert_2\cos\phi.\]

When neither norm is clamped by the numerical floor, normalizing both nonzero embeddings makes their dot product equal cosine similarity. A vector can no longer obtain a high similarity merely by being long. This can be a deliberate choice when comparing text embeddings for retrieval. It is not an instruction to normalize every hidden vector in an LM: Section 1.2 showed why magnitude can affect computation. The zero vector has no direction; the numerical convention above returns zero.

L1 normalization similarly divides by $\sum_i\lvert x_i\rvert$; max-norm scaling divides by $\max_i\lvert x_i\rvert$. L1-normalized values form probabilities only when they are nonnegative. None of these operations is the same as adding an L2 penalty to a training loss.

向量的 Euclidean length 是 $\lVert x\rVert_2=\sqrt{\sum_i x_i^2}$。这个 norm 是一个标量测量值；L2 normalization 才是利用它变换向量的操作：

\[u=\frac{x}{\max(\lVert x\rVert_2,\epsilon)}.\]

对于 $x=(3,4)$，长度为 5，结果为 $(0.6,0.8)$。对于 $(30,40)$，结果也一样。除非长度小到碰到数值下限，输出长度为 1、方向保持不变。这里不减 mean，所以 $(3,4)$ 与 $(103,104)$ 一般方向不同，归一化后也不同。

Embedding 为什么会用到它？因为 dot product 同时依赖长度与方向：

\[x^\top y=\lVert x\rVert_2\lVert y\rVert_2\cos\phi.\]

在两个 norm 都未触及数值下限时，把两个非零 embeddings 都归一化后，dot product 就等于 cosine similarity。某个向量不能仅仅因为自身很长就获得很大的 similarity。这可以是 text embedding 检索中的主动选择，但不意味着应该把 LM 内所有 hidden vectors 都这样处理：第 1.2 节已经说明 magnitude 可以参与计算。零向量没有方向；上面的数值约定会把它映射成零。

L1 normalization 类似地除以 $\sum_i\lvert x_i\rvert$；max-norm scaling 则除以 $\max_i\lvert x_i\rvert$。只有原始数值全部非负时，L1 normalization 的结果才可直接视为概率。这些操作也都不同于在 loss 中增加 L2 penalty。

For a sequence tensor $X\in\mathbb R^{B\times T\times d}$, $B$ is batch size, $T$ is sequence length, and $d$ is embedding or hidden width. Token-wise LayerNorm chooses one vector $x=X_{b,t,:}$ and applies the standardization recipe across its $d$ coordinates:

\[\operatorname{LN}(x)_i=\gamma_i \frac{x_i-\mu(x)}{\sqrt{\sigma^2(x)+\epsilon}}+\beta_i.\]

Each token gets its own two scalar statistics. The same learned vectors $\gamma,\beta\in\mathbb R^d$ are reused across tokens and examples. No running population statistics are needed; this operation uses the current input during both training and inference. This is the sequence convention used here; more generally, LayerNorm can reduce over a specified set of trailing dimensions. LayerNorm paper

The choice of axis expresses a modeling decision. We want to stabilize the representation presented by each token, even if the batch contains one example or generation has reached only one position. Computing statistics over tokens instead would couple this token’s output to other positions and potentially future information.

For example, $(1,2,3)$ and $(101,102,103)$ become identical before the affine transform. A fixed $\beta$ can restore a useful learned baseline, but it cannot reconstruct which input originally had offset 100: that input-specific information has been discarded on this branch.

对于 sequence tensor $X\in\mathbb R^{B\times T\times d}$，$B$ 是 batch size，$T$ 是 sequence length，$d$ 是 embedding 或 hidden dimension。逐 token 的 LayerNorm 选出向量 $x=X_{b,t,:}$，沿其中的 $d$ 个 coordinates 执行第 1 节的标准化：

\[\operatorname{LN}(x)_i=\gamma_i \frac{x_i-\mu(x)}{\sqrt{\sigma^2(x)+\epsilon}}+\beta_i.\]

每个 token 都有自己的两个标量统计量。相同的 learned vectors $\gamma,\beta\in\mathbb R^d$ 则在 tokens 与 examples 之间复用。这里不需要保存总体统计量，训练和推理都使用当前输入计算。这是本文采用的 sequence convention；更一般的 LayerNorm 可以沿指定的一组末尾维度计算。LayerNorm 原论文

选择这个 axis 是一个建模决定：我们希望每个 token 交给下一层的表示都有稳定尺度，即使 batch 只有一个 example，或生成只到达某个位置。如果改为跨 tokens 计算统计量，该 token 的输出就会依赖其他位置，甚至可能依赖未来信息。

例如 $(1,2,3)$ 与 $(101,102,103)$ 在 affine transform 前会变得完全相同。固定的 $\beta$ 可以提供有用的 learned baseline，却无法判断哪个输入原本多了 100：这一条分支已经丢掉了该输入特有的 offset。

We can control magnitude without first removing the mean. Consider one token’s hidden vector $x=(x_1,\ldots,x_d)$. Here $d$ is the number of coordinates in this vector, also called its hidden dimension. If the full tensor has shape $[B,T,d]$, we fix one example and one token and use only its last-axis vector. Thus $d$ is the group size $m$ from Section 1.3 in this setting, not the batch size or context length.

Root mean square means exactly “square the entries, average, then take the square root.” There are $d$ squared entries, so averaging means summing them and dividing by $d$:

\[\operatorname{RMS}(x)=\sqrt{\frac1d\sum_{j=1}^{d}x_j^2},\qquad \operatorname{RMSNorm}(x)_i=\gamma_i\frac{x_i}{\sqrt{\frac1d\sum_{j=1}^{d}x_j^2+\epsilon}}.\]

The index $j$ runs over all coordinates when computing the shared denominator; $i$ identifies the output coordinate being computed. For $x=(3,4)$, $d=2$, so $\operatorname{RMS}(x)=\sqrt{(3^2+4^2)/2}=\sqrt{12.5}$. Both coordinates are divided by the same RMS scale before their respective learned gains are applied.

The usual RMSNorm formulation learns a scale vector and omits an additive bias. It uses the same token group as token-wise LN, but no centering. It requires fewer kinds of reductions; actual speed depends on the implementation and hardware. RMSNorm paper

The relationship to L2 normalization follows directly from replacing a sum with an average:

\[\operatorname{RMS}(x)=\frac{\lVert x\rVert_2}{\sqrt d}, \qquad \frac{x}{\operatorname{RMS}(x)} =\sqrt d\,\frac{x}{\lVert x\rVert_2}.\]

Ignoring $\epsilon$ and learned scale, L2 normalization targets length 1; RMS normalization targets length $\sqrt d$, so the mean squared coordinate is 1. For $(3,4)$, RMS is $\sqrt{12.5}\approx3.536$, giving approximately $(0.849,1.131)$.

Another useful identity is

\[\operatorname{RMS}(x)^2=\sigma^2(x)+\mu(x)^2.\]

If the mean is zero, LN and RMS normalization have the same denominator and numerator before affine parameters. If the mean is large, their behavior differs. For $(101,102,103)$, RMS normalization produces three positive values close to 1; LN reveals the centered pattern $(-1.225,0,1.225)$. RMSNorm preserves direction before learned coordinate scaling; LN generally changes it by centering.

ScaleNorm uses one learned scalar $g$: $g\,x/\lVert x\rVert_2$. It learns the common target length rather than a separate gain for every coordinate. Fixed-length embedding normalization is also called FixNorm in this line of work. ScaleNorm and FixNorm

我们也可以只控制 magnitude，不先减 mean。仍然考虑一个 token 的 hidden vector $x=(x_1,\ldots,x_d)$。这里的 $d$ 就是这个向量中 coordinates 的个数，也叫 hidden dimension。如果完整 tensor 的 shape 是 $[B,T,d]$，我们固定一个 example、一个 token，只取最后一维的向量。因此在这里，$d$ 就对应第 1.3 节中的统计组大小 $m$，不是 batch size，也不是 context length。

Root mean square 的名字直接描述了计算顺序：平方、取平均、再开平方。因为有 $d$ 个数，平方后取平均，就是把这 $d$ 项加起来再除以 $d$：

\[\operatorname{RMS}(x)=\sqrt{\frac1d\sum_{j=1}^{d}x_j^2},\qquad \operatorname{RMSNorm}(x)_i=\gamma_i\frac{x_i}{\sqrt{\frac1d\sum_{j=1}^{d}x_j^2+\epsilon}}.\]

下标 $j$ 遍历全部 coordinates，用来计算共享的分母；$i$ 表示当前正在计算哪个 output coordinate。例如 $x=(3,4)$ 时，$d=2$，所以 $\operatorname{RMS}(x)=\sqrt{(3^2+4^2)/2}=\sqrt{12.5}$。两个 coordinates 使用同一个 RMS 尺度作分母，再分别乘以各自的 learned gain。

通常的 RMSNorm 学习 scale vector，不包含 additive bias。它与逐 token LN 选择同一组数，但不做 centering；需要的 reduction 种类更少，实际速度还取决于实现和硬件。RMSNorm 原论文

把“求和”换成“求平均”，就能直接看出它与 L2 normalization 的关系：

\[\operatorname{RMS}(x)=\frac{\lVert x\rVert_2}{\sqrt d}, \qquad \frac{x}{\operatorname{RMS}(x)} =\sqrt d\,\frac{x}{\lVert x\rVert_2}.\]

忽略 $\epsilon$ 和 learned scale，L2 normalization 把长度变成 1；RMS normalization 把长度变成 $\sqrt d$，使每个 coordinate 的平均平方为 1。对于 $(3,4)$，RMS 为 $\sqrt{12.5}\approx3.536$，输出约为 $(0.849,1.131)$。

另一个有用的恒等式是：

\[\operatorname{RMS}(x)^2=\sigma^2(x)+\mu(x)^2.\]

如果 mean 为零，LN 与 RMS normalization 在 affine parameters 之前具有相同的分子与分母。如果 mean 很大，行为就不同。对于 $(101,102,103)$，RMS normalization 得到三个接近 1 的正数；LN 则得到去掉共同水平后的 $(-1.225,0,1.225)$。在 learned coordinate scaling 之前，RMSNorm 保留原始方向；LN 的 centering 一般会改变方向。

ScaleNorm 只用一个 learned scalar $g$：$g\,x/\lVert x\rVert_2$。它学习一个共同的目标长度，而不是为每个 coordinate 学习不同 gain。在这条研究路线中，把 embedding 固定到指定长度也称为 FixNorm。ScaleNorm 与 FixNorm

The figure below starts with $(1,2,3)$ and lets you change its common scale and offset. Each panel plots three coordinate values, before or after the named normalization; learned affine parameters are omitted. Gray bars always show the result for the original $(1,2,3)$, while colored bars show the result for the current input. Thus a colored bar remaining level with its gray neighbor means that coordinate has not changed.

First click “Add 100 only”: the input increases, but LN still matches its gray reference; L2 and RMSNorm change. Then click “Multiply by 10 only”: all three normalized outputs almost match their references, except for the small effect of $\epsilon$. The three output panels share a fixed vertical scale, while the input panel uses its own adaptive scale—read its ticks rather than comparing its bar heights directly with output bars.

Why does normalization also change gradients? (Click to expand)

The mean and denominator are computed from the input, so backpropagation must differentiate through them. Changing one coordinate changes the denominator and therefore affects other normalized coordinates.

For pure L2 normalization, write $u=x/\lVert x\rVert_2$. At nonzero $x$, with no numerical floor active, its derivative matrix is

\[J=\frac1{\lVert x\rVert_2}(I-uu^\top).\]

$J$ maps a small input perturbation to the corresponding first-order output change. The term $uu^\top$ extracts the component parallel to the current vector. Subtracting it removes that component: increasing only the length does not change the normalized output. Transverse changes, which rotate the direction, do affect the result.

LN additionally removes the common-shift direction. These operations reshape gradient flow; they do not guarantee that every network gradient has a safe magnitude. A very small denominator can amplify sensitivity. Basics of Optimizers explains how the resulting gradient is subsequently converted into a parameter update.

下面的图从 $(1,2,3)$ 出发，可以调整共同的 scale 与 offset。四个面板分别画出输入、L2、LayerNorm、RMSNorm 输出的三个 coordinate 数值，暂时省略 learned affine parameters。灰柱始终表示原始 $(1,2,3)$ 在该面板中的结果，彩柱表示当前输入在该面板中的结果。因此，彩柱与旁边的灰柱等高，就说明这个 coordinate 没有变化。

先点“只加 100”：输入数值变大，但 LN 的彩柱仍与灰柱等高，L2 与 RMSNorm 则会变化。再点“只放大 10 倍”：除了 $\epsilon$ 的小影响，三种归一化输出都与各自的灰色参照一致。三个输出面板共用固定纵轴；输入面板则独立缩放，需要读取刻度，不能把它的柱高直接与输出面板比较。

Normalization 为什么也会改变 gradient？（点击展开）

Mean 和分母都由输入计算而来，所以 backpropagation 必须穿过这些计算求导。改变一个 coordinate 会改变分母，从而影响其他 normalized coordinates。

对于纯 L2 normalization，记 $u=x/\lVert x\rVert_2$。当 $x$ 非零且未触及数值下限时，它的导数矩阵是：

\[J=\frac1{\lVert x\rVert_2}(I-uu^\top).\]

$J$ 把微小输入变化映射成一阶近似的输出变化。$uu^\top$ 提取与当前向量平行的分量；减去它，就去掉了这个分量：如果只增加长度，normalized output 不会改变。垂直于当前方向的变化会改变方向，因此会影响结果。

LN 还会去掉共同平移的方向。这些操作改变了 gradient 如何流动，但不保证整个 network 的每个 gradient 都具有安全大小。分母很小时，敏感性仍然可能被放大。Basics of Optimizers 解释了 optimizer 随后如何把 gradient 转换成实际 parameter update。

For a causal LM, an axis choice also determines which information a token can access. Token-wise LN/RMSNorm use only that position’s coordinates, so they do not introduce access to later positions. The next methods show what changes when statistics cross tokens or examples. Image-based variants are included to interpret vision components and to make the axis comparison precise.

对于 causal LM，选择统计轴还决定了一个 token 能接触到哪些信息。逐 token LN/RMSNorm 只使用该位置的 coordinates，因此不会引入对后续位置的访问。接下来的方法说明，跨 tokens 或 examples 统计时会发生什么；图像中的变体则帮助我们理解 vision components，并准确比较 axes。

Suppose a batch contains vectors $(1,10)$, $(3,20)$, and $(5,30)$. LN compares the two entries within each row. BN compares the first coordinate across the three rows, and separately the second coordinate. The column means are $(3,20)$. Ignoring $\epsilon$ and affine parameters, the BN outputs are approximately $(-1.225,-1.225)$, $(0,0)$, and $(1.225,1.225)$.

Why compare columns? A coordinate represents the same learned feature across examples, so BN estimates that feature’s typical level and variability. For an image tensor $[B,C,H,W]$, it pools over $B,H,W$ independently for each channel $C$. The channel’s affine parameters are shared over examples and positions. BatchNorm paper

For sequence data written as $[B,T,d]$, a common temporal BN convention pools over $B,T$ per coordinate. Pooling over the full sequence during autoregressive training can leak future information through the statistics. This is one reason to distinguish “supports tensors of this shape” from “preserves this model’s causal structure.”

Training normally uses current-batch statistics and maintains running estimates. Evaluation normally uses those saved estimates, so one example’s prediction no longer depends on its evaluation companions. In PyTorch, training normalization uses the biased batch variance, while the running variance update uses an unbiased estimate; turning off running-stat tracking changes evaluation behavior. BatchNorm1d documentation

Small or correlated batches provide noisy statistics. Batch size 1 is not automatically undefined for image BN because spatial positions still contribute, but many correlated pixels are not equivalent to many independent images. Gradient accumulation also does not merge the normalization statistics of separate forward passes.

假设一个 batch 包含 $(1,10)$、$(3,20)$、$(5,30)$ 三个向量。LN 比较每一行内部的两个数；BN 则跨三行比较第一个 coordinate，再独立比较第二个 coordinate。两列 mean 是 $(3,20)$。忽略 $\epsilon$ 与 affine parameters，BN 输出约为 $(-1.225,-1.225)$、$(0,0)$、$(1.225,1.225)$。

为什么按列比较？因为同一个 coordinate 在不同 examples 中代表同一种 learned feature，BN 希望估计该 feature 的典型水平与变化范围。对于图像 tensor $[B,C,H,W]$，它为每个 channel $C$ 独立汇集 $B,H,W$；该 channel 的 affine parameters 则在 examples 与 positions 之间共享。BatchNorm 原论文

对于写成 $[B,T,d]$ 的 sequence data，一种常见 temporal BN 沿 $B,T$ 汇集每个 coordinate 的统计量。自回归训练如果跨完整 sequence 计算，就可能通过统计量泄漏未来信息。因此，“能接收这种 shape”与“保持模型的因果结构”是两件事。

训练通常使用当前 batch 的统计量，并维护 running estimates；evaluation 通常使用保存的估计量，使一个 example 的预测不依赖同时参与评估的其他 examples。PyTorch 训练时标准化用 biased batch variance，更新 running variance 时用 unbiased estimate；关闭 running-stat tracking 还会改变 evaluation 行为。BatchNorm1d 文档

较小或高度相关的 batch 会带来 noisy statistics。图像 BN 的 batch size 为 1 时不一定无法计算，因为还有 spatial positions；但许多相关 pixels 并不等于许多独立 images。Gradient accumulation 也不会自动把多次 forward pass 的 normalization statistics 合并。

InstanceNorm takes one example and one channel, then standardizes that channel over spatial positions. For $[B,C,H,W]$, its group is $H\times W$ at fixed $(b,c)$. It can remove an image’s channel-specific contrast and offset, which is useful in style transfer but may discard meaningful absolute intensity. The usual instance-statistics configuration uses each input’s statistics in both phases. InstanceNorm

GroupNorm puts $C/G$ channels into each of $G$ groups within an example and pools over those channels and spatial positions. More channels contribute to each estimate than in IN, while other examples remain excluded. Require $G$ to divide $C$. GroupNorm

At $G=C$, GN has IN’s statistical groups. At $G=1$, it normalizes the entire example over $C,H,W$. This equals LN’s statistics only if LN is configured over those same axes. It is not the same as normalizing channels independently at every pixel or token. Affine parameter shapes may still differ even when reduction groups match.

InstanceNorm 固定一个 example 和一个 channel，沿该 channel 的 spatial positions 标准化。对于 $[B,C,H,W]$，统计组就是固定 $(b,c)$ 后的 $H\times W$。它能去掉每张图像各 channel 的 contrast 与 offset，适用于 style transfer，但也可能丢掉有用的绝对强度。通常使用 instance statistics 的配置，在训练和推理时都对当前输入计算统计量。InstanceNorm

GroupNorm 在一个 example 内把 $C$ 个 channels 分成 $G$ 组，每组包含 $C/G$ 个 channels，再跨这些 channels 与 spatial positions 统计。相比 IN，每次估计使用更多 channels 的信息，同时仍不读取其他 examples。需要 $G$ 能整除 $C$。GroupNorm

当 $G=C$，GN 与 IN 的统计分组相同。当 $G=1$，它沿整个 example 的 $C,H,W$ 标准化。只有 LN 也被配置为沿这组 axes 计算时，二者统计量才一致；它不同于在每个 pixel 或 token 内单独沿 channels 计算。即使 reduction groups 相同，affine parameters 的 shapes 也仍可能不同。

The figure uses $[B,T,D]$. For images, interpret $T$ as flattened spatial positions and $D$ as channels. For sequences, $T$ is token position and $D$ is hidden width. Orange points form one reduction group. Rotate the figure: a line means one axis varies; a plane means two axes vary. The geometry describes membership in a group, not the shape of the normalization formula or the output tensor.

图中使用 $[B,T,D]$。对于图像，把 $T$ 理解为 flatten 后的 spatial positions，$D$ 理解为 channels；对于序列，$T$ 是 token position，$D$ 是 hidden width。橙色点组成一个 reduction group。旋转图可以看到：一条线表示一个 axis 在变化，一个面表示两个 axes 在变化。这个几何形状描述的是哪些元素被归到同一组，并不代表公式或输出 tensor 的形状。

SyncBatchNorm synchronizes training statistics across participating devices, increasing the group beyond one device’s local batch. It adds communication and retains BN’s dependence on which examples contribute. Frozen BN keeps saved statistics fixed, commonly when transferring a pretrained vision backbone; whether its affine parameters are also frozen is a separate choice. SyncBatchNorm documentation

Ghost BatchNorm goes the other way: compute statistics on smaller virtual sub-batches inside a large batch. The optimizer can still aggregate gradients over the large batch. Batch Renormalization adds bounded corrections based on running statistics to reduce the discrepancy between training’s batch-dependent normalization and inference. Neither operation is equivalent to LN. Ghost BN, Batch Renormalization

There are also methods that learn how to combine normalization statistics, such as Switchable Normalization, and methods combining batch and channel normalization, such as Batch-Channel Normalization. Their defining extra decision is how multiple statistical references are mixed or composed. Switchable Normalization, BCN

SyncBatchNorm 在参与训练的 devices 之间同步统计量，使统计组大于单设备的 local batch。它需要通信，也仍然依赖哪些 examples 参与统计。Frozen BN 固定保存的统计量，常见于迁移 pretrained vision backbone；affine parameters 是否也冻结，是另外一个选择。SyncBatchNorm 文档

Ghost BatchNorm 反向操作：把一个大 batch 分成较小的 virtual sub-batches，各自计算统计量；optimizer 仍可汇集整个大 batch 的 gradients。Batch Renormalization 则用 running statistics 引入有界修正，减小训练时 batch-dependent normalization 与推理之间的差异。它们都不等同于 LN。Ghost BN、Batch Renormalization

还有学习如何混合多种统计量的 Switchable Normalization，以及结合 batch 与 channel normalization 的 Batch-Channel Normalization。这些方法额外需要决定的是，多种统计参照如何被混合或组合。Switchable Normalization、BCN

In a causal text LM, the prefix already conditions each hidden state through attention. AdaLN adds another possible route: a conditioning vector explicitly chooses feature scales and shifts. That vector could describe an externally supplied task or come from an available prefix. It must not include unseen future tokens. The diffusion examples below explain the method’s established setting; AdaLN is an architectural option, not a required component of next-token prediction.

在 causal text LM 中，prefix 已经通过 attention 对 hidden states 产生条件作用。AdaLN 提供另一条可选路径：由 conditioning vector 显式决定 feature scales 与 shifts。这个向量可以描述外部指定的任务，也可以来自已经可见的 prefix，但不能包含尚不可见的未来 tokens。下面的 diffusion 例子解释该方法已有的应用场景；AdaLN 是架构选择，并不是 next-token prediction 必须具备的组件。

Ordinary LN uses the same learned $\gamma,\beta$ for every example. Suppose a generative model must respond differently to a noise level or a class label. Let $c$ be a vector encoding that condition. A small network can turn $c$ into the scale and shift:

\[s(c),b(c)=\operatorname{MLP}(c),\qquad \operatorname{AdaLN}(x,c)=(1+s(c))\odot\operatorname{LN}_0(x)+b(c).\]

$\operatorname{LN}_0$ denotes LN without learned affine parameters. The symbol $\odot$ means elementwise multiplication. The $1+s$ convention makes $s=0$ correspond to unit scale; a formulation using $\gamma(c)$ directly is equivalent after changing parameterization.

For $x$ of shape $[B,T,d]$ and global conditions $c$ of shape $[B,d_c]$, the predicted $s,b$ have shape $[B,d]$ and are broadcast over tokens. Different examples may receive different modulations, while all tokens of one example share that modulation. Input statistics are still computed separately per token.

For a simple example, suppose two normalized coordinates are $(-1,1)$. One condition predicts $s=(0,0),b=(0,0)$ and leaves them unchanged. Another predicts $s=(1,0),b=(0,3)$ and produces $(-2,4)$. The condition can amplify a feature and change its baseline before the next computation. This is conditional feature modulation; it does not alter which axes LN reduces. DiT paper

普通 LN 对每个 example 都使用同一组 learned $\gamma,\beta$。如果生成模型需要根据 noise level 或 class label 改变行为，怎么办？把条件编码成向量 $c$，再用一个小 network 从 $c$ 预测 scale 与 shift：

\[s(c),b(c)=\operatorname{MLP}(c),\qquad \operatorname{AdaLN}(x,c)=(1+s(c))\odot\operatorname{LN}_0(x)+b(c).\]

$\operatorname{LN}_0$ 是不带 learned affine parameters 的 LN，$\odot$ 表示逐元素乘法。写成 $1+s$，是为了让 $s=0$ 对应单位 scale；直接使用 $\gamma(c)$ 的写法，只是另一种参数化。

如果 $x$ 的 shape 是 $[B,T,d]$，全局条件 $c$ 的 shape 是 $[B,d_c]$，预测出的 $s,b$ 就是 $[B,d]$，然后广播到 tokens。不同 examples 可以获得不同 modulation；同一个 example 的所有 tokens 共享这组 modulation。输入的统计量仍然逐 token 独立计算。

举一个简单例子：normalized coordinates 为 $(-1,1)$。第一个 condition 预测 $s=(0,0),b=(0,0)$，结果不变；第二个 condition 预测 $s=(1,0),b=(0,3)$，结果就变成 $(-2,4)$。条件因此可以在下一步计算前放大某个 feature、调整其 baseline。这种 conditional feature modulation 不改变 LN 沿哪些 axes 统计。DiT 论文

Scale and shift control the input to a sublayer. We can separately control how much of its output is added to the residual stream:

\[h=x+\alpha(c)\odot F\!\left(\operatorname{AdaLN}(x,c)\right).\]

The gate $\alpha(c)$ is another condition-dependent vector. If it starts at zero, this sublayer initially returns $h=x$. In a DiT-style block, attention and FFN have separate scale, shift, and gate vectors: six vectors altogether. Zero-initializing the final modulation projection gives zero shifts, zero scale deviations, and zero gates. This is the relevant zero initialization, not setting every attention and FFN weight to zero. DiT reference implementation

There is a useful gradient consequence. On the first step, a zero gate suppresses the loss gradient into that residual branch’s internal weights. The gate can still receive a gradient proportional to the branch output; once it moves away from zero, the branch can learn through it. This provides a gradual way to introduce residual changes.

Scale 与 shift 控制 sublayer 收到的输入。我们还可以单独控制 sublayer 的输出有多少被加回 residual stream：

\[h=x+\alpha(c)\odot F\!\left(\operatorname{AdaLN}(x,c)\right).\]

Gate $\alpha(c)$ 是另一组由 condition 决定的向量。若它初始为零，该 sublayer 最初就返回 $h=x$。DiT-style block 的 attention 与 FFN 分别具有 scale、shift、gate，共六个向量。把最后的 modulation projection 零初始化，就能让 shifts、scale deviations 与 gates 都从零开始。这里零初始化的是这层 projection，而不是所有 attention 与 FFN weights。DiT 参考实现

这也带来一个有用的 gradient 性质：第一步中，零 gate 会抑制 loss 对 residual branch 内部 weights 的 gradient；但 gate 本身仍能收到与 branch output 成比例的 gradient。等 gate 离开零后，branch 就可以通过它学习。这样，residual change 可以逐渐加入。

FiLM is the general featurewise affine modulation $\gamma(c)\odot x+\beta(c)$. Normalization is optional: FiLM itself does not require subtracting a mean or dividing by a variance. AdaLN combines this conditioning idea with LN. Conditional BatchNorm combines condition-dependent affine parameters with BN’s statistics. An adaptive RMS variant similarly combines condition-dependent gains, and optionally shifts, with an RMS-normalized input; the actual formula should be stated because naming conventions vary. FiLM

AdaIN obtains a target channel mean and standard deviation directly from style features $s$, then applies them to standardized content features $x$:

\[\operatorname{AdaIN}(x,s) =\sigma(s)\odot\frac{x-\mu(x)}{\sqrt{\sigma^2(x)+\epsilon}}+\mu(s).\]

Each statistic is computed over spatial positions within a channel. This connects style control to matching feature statistics; with positive $\epsilon$, matching is approximate. AdaIN

SPADE lets the modulation vary across spatial locations, predicted from a semantic layout. A global class condition might ask for “a street”; a spatial condition can indicate where the road and buildings belong. Its scale and shift maps therefore retain position indices rather than being broadcast uniformly over the image. SPADE

FiLM 是一般的 featurewise affine modulation：$\gamma(c)\odot x+\beta(c)$。Normalization 是可选步骤，FiLM 本身并不要求先减 mean 或除以 variance。AdaLN 把这种 conditioning 思路与 LN 结合；Conditional BatchNorm 则把由条件决定的 affine parameters 与 BN statistics 结合。Adaptive RMS 变体类似地对 RMS-normalized input 使用条件化 gains，并可选地加入 shifts；命名约定可能不同，因此应写清实际公式。FiLM

AdaIN 直接从 style features $s$ 提取目标 channel mean 与 standard deviation，再应用于标准化后的 content features $x$：

\[\operatorname{AdaIN}(x,s) =\sigma(s)\odot\frac{x-\mu(x)}{\sqrt{\sigma^2(x)+\epsilon}}+\mu(s).\]

各统计量沿 channel 内部的 spatial positions 计算。它把 style control 与 feature statistics 的匹配联系起来；$\epsilon>0$ 时匹配是近似的。AdaIN

SPADE 让 modulation 随 spatial location 改变，由 semantic layout 预测。全局 class condition 可以要求生成“一条街道”，spatial condition 则进一步指出道路和建筑应该在哪里。因此它的 scale 与 shift maps 保留 position indices，不会统一广播到整张图像。SPADE

LMs contain learned projections for Q/K/V, FFNs, and the vocabulary head. Controlling a projection’s weights is a different intervention from normalizing the token vector supplied to it. We first compare weight-based operations, then examine the attention-specific choices: fixed dimension scaling and normalization of individual queries and keys. These operations can coexist at different sites.

LM 中的 Q/K/V、FFN 与 vocabulary head 都包含 learned projections。控制 projection weights 与归一化交给它的 token vector，是不同位置的操作。我们先比较作用于 weights 的方法，再讨论 attention 特有的选择：固定的维度 scaling，以及对单个 queries、keys 的归一化。它们可以同时出现在不同计算位置。

So far, statistics have come from activations. We can instead change how a neuron’s weight vector is represented:

\[w=g\frac{v}{\lVert v\rVert_2}.\]

In WeightNorm, optimization learns $v$ and a scalar $g$. One parameter controls direction, the other magnitude. There is no batch statistic and no promise that the resulting activations have zero mean or unit variance. WeightNorm

Weight Standardization centers and rescales the entries within each output filter:

\[\hat w=\frac{w-\operatorname{mean}(w)} {\sqrt{\operatorname{mean}((w-\operatorname{mean}(w))^2)+\epsilon}}.\]

For a convolution, the reduction covers input channels and kernel positions for one output channel. This acts on weights; GN can separately act on the resulting activations. Their combination is useful in the micro-batch setting studied by the WS work. Weight Standardization

前面的统计量来自 activations。我们也可以改变一个 neuron 的 weight vector 如何被表示：

\[w=g\frac{v}{\lVert v\rVert_2}.\]

WeightNorm 让 optimizer 学习 $v$ 与标量 $g$：一个控制方向，一个控制 magnitude。这里没有 batch statistic，也不保证输出 activations 具有零均值或单位方差。WeightNorm

Weight Standardization 则在每个 output filter 内，对 weights 的 entries 进行 centering 与 rescaling：

\[\hat w=\frac{w-\operatorname{mean}(w)} {\sqrt{\operatorname{mean}((w-\operatorname{mean}(w))^2)+\epsilon}}.\]

对于 convolution，统计范围是一个 output channel 对应的 input channels 与 kernel positions。WS 作用于 weights，GN 则可以另外作用于输出 activations。WS 工作研究了这种组合在 micro-batch 训练中的用途。Weight Standardization

Normalizing each weight vector does not directly control how a full matrix amplifies arbitrary inputs. Its spectral norm, the largest singular value, answers that question:

\[\sigma_{\max}(W)=\max_{\lVert u\rVert_2=1}\lVert Wu\rVert_2,\qquad \bar W=\frac{W}{\sigma_{\max}(W)}.\]

If $W$ stretches one axis by 3 and another by 1, its spectral norm is 3. Dividing the whole matrix by 3 limits the largest stretch to 1; the second axis now stretches by $1/3$. This preserves relative singular values, rather than making all directions equally strong.

With exact normalization and nonzero $W$, $\lVert\bar Wx-\bar Wy\rVert_2\le\lVert x-y\rVert_2$. This is a Lipschitz bound for that linear map. Implementations usually estimate the largest singular value by power iteration; an approximate estimate is not an exact certificate. Residual additions and other layers must be accounted for when discussing the full network. For convolutions, normalizing a reshaped kernel need not give the exact norm of the full spatial convolution operator. Spectral Normalization

逐个归一化 weight vectors，并不直接限制完整矩阵如何放大任意输入。矩阵的 spectral norm，也就是最大 singular value，回答了这个问题：

\[\sigma_{\max}(W)=\max_{\lVert u\rVert_2=1}\lVert Wu\rVert_2,\qquad \bar W=\frac{W}{\sigma_{\max}(W)}.\]

如果 $W$ 沿一个轴放大 3 倍，沿另一个轴放大 1 倍，其 spectral norm 就是 3。整个矩阵除以 3 后，最大放大倍数变成 1；另一个轴则变成 $1/3$。它保留 singular values 的相对关系，不会把所有方向变成同样强。

若归一化精确且 $W$ 非零，就有 $\lVert\bar Wx-\bar Wy\rVert_2\le\lVert x-y\rVert_2$，这叫该 linear map 的 Lipschitz bound。实际实现通常通过 power iteration 估计最大 singular value；近似估计不等于严格保证。讨论完整 network 时，还需要考虑 residual additions 与其他 layers。对于 convolution，对 reshape 后的 kernel 做归一化，也不一定等于限制完整 spatial convolution operator 的精确 norm。Spectral Normalization

Yes: dividing attention scores by $\sqrt{d_k}$ is normalization in the broad sense of setting a reference scale. It is usually called scaled dot-product attention. Unlike LN or RMSNorm, the denominator is fixed by the head dimension, rather than measured from the current activations:

\[s_{ij}=\frac{q_i^\top k_j}{\sqrt{d_k}}.\]

Why does dimension enter? One dot product sums $d_k$ coordinate products. Suppose the query and key coordinates are independent, zero-mean, and unit-variance. Each product then has variance 1, and the sum has variance $d_k$. Its standard deviation, a measure of typical magnitude, is $\sqrt{d_k}$. Dividing by that number returns the variance to 1 under these assumptions. Scaled dot-product attention

For example, increasing head width from 16 to 64 increases the unscaled score’s standard deviation from 4 to 8 in this toy model. Without compensation, a wider head can produce sharper attention merely because it sums more terms. The divisor compensates for this predictable dimensional effect.

The issue is the difference between competing scores. In a two-key example, one key’s softmax weight is a sigmoid of the score gap. Very large positive or negative gaps place it in a flat tail where small score changes barely move the weight. Scaling helps avoid this source of small softmax derivatives. It does not guarantee that the full loss gradient is small or large: that also depends on the downstream loss and the rest of the computation. The sigmoid illustration makes this local sensitivity visible.

The assumptions also explain the limitation. Learned queries and keys need not retain unit variance or independence. If their magnitudes grow during training, a fixed $\sqrt{d_k}$ cannot respond. It neither forces each score row to unit variance nor makes each query a unit vector.

Why a square root, rather than dₖ? (Click to expand)

For independent zero-mean summands, variances add. The variance grows by a factor of $d_k$, so the standard deviation grows by $\sqrt{d_k}$. Dividing the score by $d_k$ instead would give variance $1/d_k$, progressively shrinking the logits as width grows. That would compute an average coordinate product, which has a different scale target.

If query and key coordinates have variances $\sigma_q^2$ and $\sigma_k^2$, the same independence calculation gives unscaled score variance $d_k\sigma_q^2\sigma_k^2$. After division by $\sqrt{d_k}$, the variance is $\sigma_q^2\sigma_k^2$, not necessarily 1. The operation cancels width, not arbitrary activation scale.

是的：从“提供一个参照尺度”的广义理解出发，attention score 除以 $\sqrt{d_k}$ 是一种 normalization，通常称为 scaled dot-product attention。它的分母由 head dimension 决定，不像 LN 或 RMSNorm 那样从当前 activations 测量：

\[s_{ij}=\frac{q_i^\top k_j}{\sqrt{d_k}}.\]

为什么 dimension 会出现？一个 dot product 把 $d_k$ 个 coordinate products 加起来。假设 query 与 key 的 coordinates 相互独立、均值为零、方差为一，每个乘积的 variance 就是 1，求和后的 variance 就是 $d_k$。描述典型 magnitude 的 standard deviation 因而是 $\sqrt{d_k}$。除以它，就在这些假设下把 variance 调回 1。Scaled dot-product attention 原论文

例如，在这个 toy model 中，head width 从 16 增到 64，unscaled score 的 standard deviation 就从 4 变成 8。如果不补偿，较宽的 head 可能仅仅因为累加了更多项，就产生更尖锐的 attention。这个 divisor 补偿的是可预测的维度效应。

真正相关的是竞争 scores 之间的差距。在两个 keys 的例子中，一个 key 的 softmax weight 是 score gap 的 sigmoid。很大的正向或负向 gap 会把它推到平坦尾部，此时 score 的小变化几乎不改变 weight。Scaling 有助于避免这种 softmax 导数变小的情况；完整 loss gradient 的大小，还依赖下游 loss 与其余计算，不能仅由这一点决定。Sigmoid 小图 展示了这种局部敏感性。

这些假设也说明了限制：训练后的 queries 与 keys 不一定保持单位方差或独立性。如果它们的 magnitude 在训练中增长，固定的 $\sqrt{d_k}$ 无法随之调整。它既不强制每一行 scores 的 variance 为 1，也不把每个 query 变成 unit vector。

为什么除以平方根，而不是 dₖ？（点击展开）

对于独立、零均值的加项，variances 可以相加。Variance 增长 $d_k$ 倍，standard deviation 就增长 $\sqrt{d_k}$ 倍。如果改为把 score 除以 $d_k$，variance 会变为 $1/d_k$，随着 width 增大而不断缩小 logits。这相当于计算 coordinate product 的平均值，目标尺度不同。

如果 query 与 key coordinates 的 variance 分别是 $\sigma_q^2$ 与 $\sigma_k^2$，同样的独立性计算给出 unscaled score variance 为 $d_k\sigma_q^2\sigma_k^2$。除以 $\sqrt{d_k}$ 后，variance 是 $\sigma_q^2\sigma_k^2$，不一定为 1。它抵消的是 width，而不是任意 activation scale。

In attention, the score $q^\top k$ depends on both vector lengths and their alignment. The original QK-Norm proposal L2-normalizes each query and key along the head dimension and uses a learned scale for the resulting cosine score:

\[s_{ij}=g\left(\frac{q_i}{\lVert q_i\rVert_2}\right)^\top \left(\frac{k_j}{\lVert k_j\rVert_2}\right).\]

The learned scale controls how sharp attention can become without requiring Q/K magnitudes to grow. This proposal replaces the usual fixed $1/\sqrt{d_k}$ score scaling. “QK normalization” can also refer to architectures applying LN or RMSNorm to Q and K; axis, affine parameters, and subsequent score scaling must be specified. Original QK-Norm

Softmax comes after these scores and a visibility mask. It converts allowed scores into nonnegative weights that sum to 1. L2-normalizing a key vector does not create attention probabilities, and softmax does not standardize the Q/K coordinates. Attention and Language Models follows the full Q/K/V computation.

Attention score $q^\top k$ 同时依赖向量长度与方向匹配程度。最初的 QK-Norm 沿 head dimension 对每个 query 和 key 做 L2 normalization，再为得到的 cosine score 加 learned scale：

\[s_{ij}=g\left(\frac{q_i}{\lVert q_i\rVert_2}\right)^\top \left(\frac{k_j}{\lVert k_j\rVert_2}\right).\]

Learned scale 控制 attention 可以有多尖锐，而不必依赖 Q/K magnitudes 增长。在这项方法中，它替代通常固定的 $1/\sqrt{d_k}$ score scaling。“QK normalization” 也可能指某些架构对 Q/K 使用 LN 或 RMSNorm，因此需要明确 axis、affine parameters 与后续 score scaling。原始 QK-Norm

Softmax 在这些 scores 与 visibility mask 之后，把允许的 scores 变成非负、加和为 1 的 weights。对 key vector 做 L2 normalization 不会产生 attention probabilities；softmax 也没有标准化 Q/K coordinates。Attention and Language Models 串联了完整的 Q/K/V 计算。

Choosing the function and choosing its position are separate decisions. Let $N$ be LN or RMSNorm and $F$ an attention or FFN sublayer:

\[\text{Pre-Norm:}\quad y=x+F(N(x)), \qquad \text{Post-Norm:}\quad y=N(x+F(x)).\]

Pre-Norm gives the sublayer a normalized input while keeping the residual bypass unchanged. Post-Norm normalizes the result of the addition, so the bypass also passes through normalization. A block containing attention and FFN usually has a separate normalization site and residual addition for each sublayer. A final normalization can then precede the output head. Pre-Norm analysis

For Pre-Norm, the local derivative includes an identity term: $I+J_FJ_N$. This gives gradients a direct path, though it does not guarantee stability through an arbitrary stack. Notice also that LN discards offset only on the normalized branch; the residual path can preserve the original information. The full block therefore has different invariances from LN in isolation.

Return to $x_A=(1,2,3)$ and $x_B=10x_A$. Ignore $\epsilon$, and fix all other inputs and any randomness. LN or RMSNorm maps these two states to the same normalized input, so the sublayer computes the same update $u$. The residual outputs are nevertheless $x_A+u$ and $10x_A+u$, which remain different.

If $u$ is not parallel to $x_A$, those outputs generally have different directions as well. For example, with $u=(1,0,0)$ they are $(2,2,3)$ and $(11,20,30)$; even a subsequent LN distinguishes their centered patterns. Thus scale preserved by a residual path can influence information read by a later normalized branch. This is a possible route, not a guarantee: a final normalization can still erase distinctions that remain only a common positive scale (and, for LN, a common offset).

The distinction matters when assessing whether normalization is appropriate for an LM. Training with normalization allows upstream weights to develop useful representations under these constraints. Inserting it into an already trained model is not generally equivalent: it can change attention, residual updates, and predictions. Nor does a large norm automatically indicate greater semantic importance; its effect must be traced through the actual computation.

LayerScale, ReZero, and depth-dependent residual scaling adjust the strength of residual branches. They are related stabilization choices, but do not themselves compute an activation mean, variance, or norm. AdaLN-Zero combines modulation with such a gate. LayerScale, ReZero

选择 normalization function 与选择它的位置，是两个决定。令 $N$ 为 LN 或 RMSNorm，$F$ 为 attention 或 FFN sublayer：

\[\text{Pre-Norm:}\quad y=x+F(N(x)), \qquad \text{Post-Norm:}\quad y=N(x+F(x)).\]

Pre-Norm 给 sublayer 提供 normalized input，同时保留原样的 residual bypass。Post-Norm 则对 addition 的结果归一化，所以 bypass 也要经过 normalization。包含 attention 与 FFN 的 block，通常为每个 sublayer 分别设置 normalization 与 residual addition；output head 前还可以有 final normalization。Pre-Norm 分析

Pre-Norm 的局部导数包含 identity term：$I+J_FJ_N$。它给 gradient 提供一条直接路径，但不保证任意深度的 stack 都稳定。另外，LN 只在 normalized branch 上丢掉 offset；residual path 仍能保留原始信息。因此完整 block 的 invariances 与单独 LN 不同。

回到 $x_A=(1,2,3)$ 与 $x_B=10x_A$。忽略 $\epsilon$，固定其他输入和随机性，LN 或 RMSNorm 会把这两个 states 变成相同的 normalized input，所以 sublayer 计算出相同 update $u$。但 residual outputs 分别是 $x_A+u$ 与 $10x_A+u$，仍然不同。

如果 $u$ 不平行于 $x_A$，两个输出一般连方向都会不同。例如 $u=(1,0,0)$ 时，输出是 $(2,2,3)$ 与 $(11,20,30)$；即使接下来再做 LN，也能区分它们 centered 后的模式。因此，residual path 保留的尺度可以影响后续 normalized branch 读到的信息。这是一条可能的路径，不是保证：如果某些区别到最后仍然只是共同的正向 scale，final normalization 仍可能把它们去掉；对于 LN，共同 offset 也可能被去掉。

因此，在 LM 中判断 normalization 是否合适，需要看完整计算。带着 normalization 从头训练，让上游 weights 可以在这些约束下学习有用的表示；把它直接插进已经训练好的模型，一般不会保持等价，可能改变 attention、residual updates 与预测。同时，较大的 norm 也不自动代表更重要的语义，需要沿实际计算追踪其作用。

LayerScale、ReZero 和随深度设置的 residual scaling，调整的是 residual branches 的贡献强度。它们是相关的稳定化选择，但本身不计算 activation mean、variance 或 norm。AdaLN-Zero 把 modulation 与这样的 gate 结合起来。LayerScale、ReZero

The snippet below implements vector L2, token LN, RMSNorm, and conditional LN in PyTorch. Hover or select a highlighted expression to see its shape and role beside the code. These are transparent mathematical implementations; production kernels may fuse the same operations. For low-precision inputs, the reductions are accumulated in float32 before casting the result back.

For $[B,T,d]$, reducing over the last axis with keepdim=True produces $[B,T,1]$. Broadcasting repeats each scalar over that token’s $d$ coordinates. AdaLN instead broadcasts a condition’s $[B,d]$ modulation as $[B,1,d]$. These two singleton axes serve different purposes.

Use the centered-square variance formula to avoid subtracting two large nearly equal quantities. Treat $\epsilon$ as part of the definition: adding it inside a square root, outside a square root, or using a clamped norm are different operations near zero.

下面用 PyTorch 实现 vector L2、逐 token LN、RMSNorm 与 conditional LN。Hover 或选中高亮表达式，可以在代码旁边查看 shape 与作用。这些实现直接对应数学定义；实际 production kernel 可能融合相同操作。对低精度输入，reduction 使用 float32，最终再转换回输入 dtype。

对于 $[B,T,d]$，沿最后一维并保留 keepdim=True，得到 $[B,T,1]$；broadcasting 把每个 scalar 应用到该 token 的 $d$ 个 coordinates。AdaLN 则把 condition 的 $[B,d]$ modulation 变为 $[B,1,d]$ 再广播。这两种 singleton axis 的作用不同。

计算 variance 时先 centering 再平方，可以避免两个很大且相近的量相减。$\epsilon$ 也是定义的一部分：加在平方根里面、外面，或采用 clamped norm，在接近零时是不同操作。

Token IDs are categorical indices into an embedding table; their numerical mean and variance do not describe word meaning. The input preprocessing methods below instead concern continuous features, such as audio/image inputs to a multimodal model or embedding representations in a downstream pipeline. They should not be applied to token IDs as if those IDs were measured quantities.

Token IDs 是 embedding table 中的类别索引，它们数值的 mean 与 variance 不描述词义。下面的数据预处理针对连续 features，例如 multimodal model 的音频、图像输入，或下游流程中的 embedding representations。不能把 token IDs 当成测量值，直接用这些方法标准化。

Before a network, z-score standardization uses a training dataset’s per-feature mean and standard deviation. Save them and reuse them for validation and test data. Unlike token LN, this applies a fixed reference learned from the dataset. Min-max scaling maps training extrema to a target range; new out-of-range inputs can still fall outside it. Robust scaling uses a median and a quantile range, making the reference less sensitive to extreme values. It does not remove outliers automatically. Preprocessing reference

Whitening goes beyond independent feature scaling by transforming the covariance matrix toward identity. If the centered data have covariance $\Sigma$, a regularized symmetric whitening transform is

\[z=(\Sigma+\epsilon I)^{-1/2}(x-\mu).\]

The matrix inverse square root rotates and rescales correlated directions. Without regularization and with invertible $\Sigma$, the transformed covariance is identity. With $\epsilon>0$, this is approximate. LN only controls the statistics within its selected group; it does not perform this covariance transformation.

Network 之前的 z-score standardization，使用 training dataset 中每个 feature 的 mean 与 standard deviation；保存后应用于 validation/test data。它使用从数据集估计出的固定参照，与逐 token LN 不同。Min-max scaling 把训练数据的 extrema 映射到目标区间，新输入超出训练范围时仍可能超出该区间。Robust scaling 使用 median 与 quantile range，让参照对 extreme values 不那么敏感；它不会自动删除 outliers。数据预处理参考

Whitening 进一步把 covariance matrix 变换到接近 identity，不仅仅独立调整每个 feature 的尺度。若 centered data 的 covariance 为 $\Sigma$，一种正则化的对称 whitening 为：

\[z=(\Sigma+\epsilon I)^{-1/2}(x-\mu).\]

矩阵 inverse square root 对相关的方向进行旋转与缩放。没有 regularization 且 $\Sigma$ 可逆时，变换后的 covariance 是 identity；$\epsilon>0$ 时是近似的。LN 只控制选定组内的统计量，不执行这种 covariance transformation。

Two related terms deserve a final distinction. Gradient clipping scales a gradient down only when it exceeds a threshold; gradient normalization sets a target norm. Adam’s second-moment scaling uses an optimizer state accumulated across steps. These act on updates or gradients, not hidden activations, and are developed in Basics of Optimizers.

Other specialized methods include Local Response Normalization (local competition across neighboring channels) and FRN (per-channel spatial RMS normalization paired with a thresholded activation). Their groups and transforms differ from token RMSNorm. LRN documentation, Filter Response Normalization

When encountering another “Norm,” reconstruct its group, measurement, transform, learned parameters, and train/eval behavior. Those five pieces are enough to place it in this framework and to identify what an implementation actually does.

另有两个相关概念需要区分。Gradient clipping 只在 gradient 超过阈值时将其缩小；gradient normalization 则指定目标 norm。Adam 的 second-moment scaling 使用跨 steps 积累的 optimizer state。它们作用于 updates 或 gradients，而不是 hidden activations，详见 Basics of Optimizers。

更专门的方法包括 Local Response Normalization（相邻 channels 之间的局部竞争）与 FRN（每个 channel 的 spatial RMS normalization，再配合带阈值的 activation）。它们的 groups 与 transforms 和逐 token RMSNorm 不同。LRN 文档、Filter Response Normalization

以后遇到另一个 “Norm”，就依次找出它的分组、测量、变换、learned parameters，以及 train/eval 行为。这五部分能帮助我们把它放进这个框架，也能帮助我们读懂实现真正执行的操作。