Score Centering under Policy Mismatch
Policy-gradient methods multiply the gradient of an action’s log-probability by its reward. When the policy that generated the data differs from the policy being optimized, this update can have a nonzero expectation even if every action receives the same reward. This log-probability gradient is called the score. Score Centering (SC) subtracts its mean under the sampling policy to address this effect.
The motivating requirement is simple: if reward does not distinguish between actions, the gradient of expected reward is zero. Exact SC restores this cancellation under the sampling policy; importance sampling reweights the update to recover an expectation under the target policy. These are different properties. We use a binary-action example to explain their relationship, then discuss language-model updates. The discussion builds on Marek and Ryabinin’s Score Centering paper; the interactive examples evaluate analytic expressions.
1. Policy Mismatch and the Mean Score
1. 策略不匹配与 score 的均值
1.1 The Target Gradient and the Sampling Policy
1.1 目标梯度与采样策略
Start with one decision and finitely many possible actions. Let $\theta$ be the model parameters, $a$ a possible action, and $R(a)$ its reward, which does not depend directly on $\theta$. The current policy assigns probability $p_\theta(a)$ to action $a$. The sampling policy assigns probability $q(a)$ and generates the observed action $y\sim q$. An older checkpoint or different numerical precision can make these policies differ. We hold $q$ fixed during differentiation.
Write $p$ as shorthand for the distribution $p_\theta$. An expectation is a probability-weighted average: outcomes that occur more often contribute more. For any function $f$ of the action, $\mathbb E_p[f]=\sum_a p_\theta(a)f(a)$; $\mathbb E_q[f]$ uses $q(a)$ instead. Each sum runs over all possible actions. Our objective $J(\theta)$ is the current policy’s expected reward:
\[J(\theta)=\mathbb E_{a\sim p_\theta}[R(a)] =\sum_a p_\theta(a)R(a).\]The score $s_p(a)$ is the gradient of an action’s log-probability with respect to the parameters. Let $g_p$ denote the gradient of the reward objective:
\[\begin{aligned} s_p(a)&=\nabla_\theta\log p_\theta(a),\\ g_p&=\nabla_\theta J(\theta)=\mathbb E_p[R\,s_p]. \end{aligned}\]Inside an expectation, $R$ and $s_p$ are evaluated at the same random action. We assume these derivatives and expectations exist. The desired gradient $g_p$ weights each action’s score by both its reward and its probability under the current policy. This follows by differentiating the sum defining $J$: rewards stay fixed, and $\nabla_\theta p_\theta(a)=p_\theta(a)\nabla_\theta\log p_\theta(a)$.
A sample from $q$ instead gives the uncorrected direction $\widehat g_{\rm raw}=R(y)s_p(y)$, whose expectation is $g_{\rm raw}=\mathbb E_q[\widehat g_{\rm raw}]$. The hat marks a direction computed from one random sample; $g_{\rm raw}$ denotes its mean. Throughout the article, an “update” means an ascent direction before multiplication by the learning rate. Because $g_{\rm raw}$ averages under $q$, it generally differs from the target gradient $g_p$.
1.2 A Binary-Action Example with Constant Reward
1.2 一个恒定奖励的二元动作例子
Consider an action $y\in\lbrace 0,1\rbrace$. Write $p_1=p_\theta(1)$ and $q_1=q(1)$ for the current and sampling policies’ probabilities of action 1. Their probabilities of action 0 are $1-p_1$ and $1-q_1$. Assume both probabilities lie strictly between zero and one.
To describe a parameter update, we need a parameter that determines the action probabilities. In this example the model has one real-valued parameter $\theta$. It determines $p_1$ through
\[p_1=\sigma(\theta)=\frac{1}{1+e^{-\theta}},\qquad \theta=\log\frac{p_1}{1-p_1}.\]The ratio $p_1/(1-p_1)$ is the odds of action 1 relative to action 0. Its natural logarithm is the logit. Equal probabilities give odds one and logit zero; a positive logit favors action 1, and a negative logit favors action 0. Increasing $\theta$ increases $p_1$. The function $\sigma$, called the logistic function, converts the unconstrained real parameter into a probability. We update $\theta$, while the sampling probability $q_1$ stays fixed.
The score tells us how the sampled action’s log-probability changes when $\theta$ increases. Since $\mathrm d p_1/\mathrm d\theta=p_1(1-p_1)$, the chain rule gives
\[\begin{aligned} s_p(1)&=\frac{\mathrm d\log p_1}{\mathrm d\theta} =\frac{1}{p_1}p_1(1-p_1)=1-p_1,\\ s_p(0)&=\frac{\mathrm d\log(1-p_1)}{\mathrm d\theta} =-\frac{1}{1-p_1}p_1(1-p_1)=-p_1. \end{aligned}\]The signs agree with the probability changes: increasing $\theta$ makes action 1 more likely and action 0 less likely. Both cases can be written as $s_p(y)=y-p_1$.
We can now compute the average score under the current policy. Averaging means weighting each score by how often its action is sampled. For $p_1=0.6$, action 1 has score $0.4$ and occurs with probability $0.6$; action 0 has score $-0.6$ and occurs with probability $0.4$. The contributions are $0.24$ and $-0.24$. In general,
\[\mathbb E_{y\sim p_\theta}[s_p(y)] =p_1(1-p_1)+(1-p_1)(-p_1)=0.\]The positive score occurs more often in this example, but the less frequent negative score has a larger magnitude. Their probability-weighted contributions balance exactly. Now sample with $q_1=0.8$ while keeping the current probability $p_1=0.6$. The scores remain $0.4$ and $-0.6$, but their frequencies change: the average is $0.8\cdot0.4+0.2\cdot(-0.6)=0.2$. In symbols,
\[\mathbb E_{y\sim q}[s_p(y)] =q_1(1-p_1)+(1-q_1)(-p_1)=q_1-p_1.\]Finally, give both actions reward one: $R(0)=R(1)=1$. The objective $J(\theta)$ is the current policy’s expected reward, so
\[J(\theta)=\mathbb E_{y\sim p_\theta}[R(y)] =p_1\cdot1+(1-p_1)\cdot1=1.\]Changing the action probabilities cannot change this reward. Thus $J(\theta)$ has gradient zero. Yet the raw expected update remains $g_{\rm raw}=0.2$: with a positive learning rate it increases $\theta$ and hence $p_1$, without improving expected reward. This is an example of the drift term defined in Section 2.2.
The zero mean under the current policy extends beyond two actions. Multiplying a log-probability gradient by its probability gives the probability gradient itself. These gradients sum to zero because the probabilities always sum to one:
\[\mathbb E_p[s_p] =\sum_a p_\theta(a)\nabla_\theta\log p_\theta(a) =\nabla_\theta\sum_a p_\theta(a) =\nabla_\theta 1=0.\]This calculation assumes positive differentiable action probabilities. Using a different distribution $q$ as the weights no longer gives the derivative of that normalized sum.
2. What Score Centering Corrects
2. Score Centering 修正了什么
2.1 Centering under the Sampling Policy
2.1 在采样策略下对 score 中心化
We can derive the centering correction from the constant-reward requirement, rather than introduce it as a formula. Keep the single-decision objective from Section 1; any state or prefix is held fixed and omitted from the notation. If every action has reward $R(a)=1$, then $J(\theta)=\mathbb E_{a\sim p_\theta}[R(a)]=1$. Changing any model parameter cannot improve this objective, so its gradient must be zero in every parameter direction.
The raw update uses the current policy’s score $s_p(a)=\nabla_\theta\log p_\theta(a)$, but the action is drawn from $q$. With constant reward, its expectation is
\[\mathbb E_{a\sim q}[R(a)s_p(a)]=\mathbb E_{a\sim q}[s_p(a)],\]which need not be zero when the sampling and current policies differ. Even though the reward expresses no preference between actions, the uncorrected update can have a systematic direction.
To remove that direction, consider subtracting the same vector $c$ from every sampled action’s score. The vector may depend on the fixed policies and context, but not on which action is drawn. For constant reward, the desired condition is
\[\mathbb E_{a\sim q}[s_p(a)-c] =\mathbb E_{a\sim q}[s_p(a)]-c=0.\]Thus, among such action-independent subtractions, the required choice is exactly the sampling-policy mean:
\[c=\bar s=\mathbb E_{a\sim q}[s_p(a)].\]This gives SC. For an observed action $y\sim q$ and a general reward $R(y)$, its single-sample update is
\[\widehat g_{\rm SC}=R(y)\bigl(s_p(y)-\bar s\bigr).\]The bar denotes the score mean; the hat denotes a sampled update. With an exact $\bar s$, constant reward gives zero expected update. This does not require every individual sample or finite batch to produce zero. In the binary example, subtracting $\bar s=q_1-p_1$ from $y-p_1$ gives $y-q_1$, whose mean is zero because $\mathbb E_q[y]=q_1\cdot1+(1-q_1)\cdot0=q_1$.
Matching the zero gradient for constant reward does not establish unbiasedness for general rewards. For $R=1$, exact SC does recover the correct gradient in expectation. When rewards vary with the action, however, the sample frequencies still come from $q$, and centering alone need not recover the gradient under $p_\theta$. Sections 2.2–2.3 make this distinction explicit.
2.2 Covariance and Reward Offsets
2.2 协方差与奖励平移
The covariance between reward and score measures their joint variation under the same sampled action. For a vector-valued score, it is computed component by component:
\[\operatorname{Cov}\nolimits_{q}(R,s_p) =\mathbb E_q\bigl[(R-\mathbb E_q[R])(s_p-\bar s)\bigr].\]For one parameter component, an action with both above-average reward and above-average score contributes positively; one with opposite deviations contributes negatively. The covariance averages these contributions using the action probabilities under $q$. Here $\bar s=\mathbb E_q[s_p]$ is the mean defined in Section 2.1. Expanding this expression separates the raw expected update into two terms:
\[\begin{aligned} \mathbb E_q[R\,s_p] &=\mathbb E_q[R]\,\bar s+\operatorname{Cov}\nolimits_{q}(R,s_p),\\ \mathbb E_q[\widehat g_{\rm SC}]&=\operatorname{Cov}\nolimits_{q}(R,s_p). \end{aligned}\]The drift term $\mathbb E_q[R]\bar s$ depends on the mean reward and mean score. The covariance term describes how reward differences across actions are associated with score differences. SC removes the former and retains the latter. Adding a constant to every reward therefore leaves the SC expectation unchanged, while changing the raw expectation by that constant times $\bar s$.
A reward baseline is a value subtracted from the reward before weighting the score. Taking the action-independent value $b=\mathbb E_q[R]$ gives the same expected covariance in this single-decision setting, but the sampled updates $R(s_p-\bar s)$ and $(R-b)s_p$ generally differ. This $b$ is the exact population mean, not a mean estimated from a finite group of responses; Section 4.2 discusses group reward centering.
2.3 The Scope of the Correction
2.3 中心化修正的范围
The SC paper, Section 4, distinguishes the covariance under the sampler from the covariance under the current policy. After centering, the expected update remains $\operatorname{Cov}\nolimits_{q}(R,s_p)$; the target gradient is $\operatorname{Cov}\nolimits_{p}(R,s_p)$, since $\mathbb E_p[s_p]=0$. An estimate is unbiased for $g_p=\nabla_\theta J$ when its expectation equals that target gradient. These two covariances generally differ when $p\ne q$: SC subtracts a mean, but the actions still occur with frequencies determined by $q$, so their reward–score associations are still averaged with those frequencies. SC therefore restores the zero-mean score property without generally recovering an unbiased target-gradient estimate. When the sampling and current policies agree, $p=q$, the setting is called on-policy. The exact correction is zero and SC leaves the on-policy update unchanged.
3. Score Centering and Importance Sampling
3. Score Centering 与 Importance Sampling
3.1 Correcting the Action Distribution
3.1 修正动作分布
Importance sampling (IS) assigns each sampled action the weight $w(a)=p(a)/q(a)$. When an action is less probable under $q$ than under $p$, its contribution receives a larger weight; the reverse holds when it is more probable. For the binary example, action 1 accounts for probability $0.8$ under the sampler; multiplying by $0.6/0.8$ gives an effective contribution weight of $0.6$. Action 0 similarly gives $0.2\cdot(0.4/0.2)=0.4$. These are exactly the current policy’s probabilities. In general,
\[\begin{aligned} \mathbb E_q[wR\,s_p] &=\sum_a q(a)\frac{p(a)}{q(a)}R(a)s_p(a)\\ &=\mathbb E_p[R\,s_p]=g_p. \end{aligned}\]The support of a policy is the set of actions to which it assigns positive probability. A sufficient support condition for this identity is $q(a)>0$ whenever $p(a)>0$. If this support condition fails, an action that contributes to the target average can never appear in the sampled data; no weight on the observed actions can restore its missing contribution. We also require the weighted update to have a finite expected magnitude. Even with adequate support, large probability ratios can cause large sample-to-sample fluctuations, or high variance. Clipping replaces weights above a chosen limit by that limit. This bounds their magnitude but generally changes the expected update, introducing bias relative to $g_p$.
SC and IS therefore make different corrections. SC makes the score mean zero under the sampling policy. Exact IS converts the reward-weighted score expectation from the sampling policy to the target policy. The sampling distribution itself remains $q$ in both cases.
3.2 Centering a Weighted Score
3.2 加权 score 的中心化
The SC paper, Appendix A.2, applies centering to weighted scores. Let $u(a)$ denote a chosen action weight: it may be one, the exact IS ratio $w(a)=p(a)/q(a)$, or a clipped version of that ratio. Define $c_u$ as the mean of the weighted score under $q$, and let $\widehat g_u$ be the resulting single-sample update for $y\sim q$:
\[\begin{aligned} c_u&=\mathbb E_{a\sim q}[u(a)s_p(a)],\\ \widehat g_u&=R(y)\bigl(u(y)s_p(y)-c_u\bigr). \end{aligned}\]For fixed policies and weights, $c_u$ is the same vector whichever action is sampled. Subtracting it makes a constant reward give zero expected update. With exact weights $u=w$, the assumptions of Section 3.1 give $c_u=\mathbb E_p[s_p]=0$. Clipping, or masking selected weights to zero, can make this mean nonzero. The scalar-loss implementation in Section 4.3 treats these weights as fixed coefficients during differentiation.
Why the order of weighting and centering matters
A quantity with mean zero need not retain mean zero after multiplication by different action-dependent weights. The correction must therefore use the mean of the weighted score actually used in the update. Recall $\bar s=\mathbb E_q[s_p]$, the unweighted score mean, and $w(a)=p(a)/q(a)$, the exact IS weight. Centering the score first and then multiplying by $w$ gives
\[\mathbb E_q[wR(s_p-\bar s)] =g_p-\mathbb E_p[R]\,\bar s.\]Here $g_p=\nabla_\theta J$ is the target reward gradient from Section 1.1. The additional term is generally nonzero. Thus this order differs from centering the weighted score, and “SC + IS” must specify which estimator is used.
4. Score Centering in Language Models
4. 语言模型中的 Score Centering
4.1 Token Corrections and Sequence-Level Gradients
4.1 Token 修正与序列级梯度
A token is a unit of generated text. Fix a prompt $x$, and write a complete response as $\tau=(y_1,\ldots,y_T)$, where $y_t$ is the token at position $t$ and $T$ is the number of generated tokens. The prefix before position $t$ is $h_t=(x,y_1,\ldots,y_{t-1})$. It includes the prompt and all previously generated tokens.
At a fixed prefix $h$, the distributions $p_\theta(a\mid h)$ and $q(a\mid h)$ give the next-token probabilities. The conditional score and its sampling-policy mean are
\[\begin{aligned} s_p(a,h)&=\nabla_\theta\log p_\theta(a\mid h),\\ \bar s(h)&=\sum_a q(a\mid h)s_p(a,h). \end{aligned}\]SC subtracts $\bar s(h)$ from the sampled token’s score. Exact token IS instead weights it by $p_\theta(a\mid h)/q(a\mid h)$. Both operations concern a fixed prefix; neither alone corrects how often that prefix is reached or which tokens are generated afterward.
For example, a prefix may occur with probability $0.9$ under the current policy and $0.5$ under the sampling policy. Even if their next-token distributions at that prefix are identical, making the local ratios one, the prefix is still underrepresented in the sampled data. If the reward is assigned only after completing the response, the expected reward can also depend on which policy generates the remaining tokens.
The objective $J$ from Section 1.1 now applies to complete responses: $J(\theta)=\mathbb E_{\tau\sim p_\theta}[R(\tau)]$, where $R(\tau)$ is the reward for the whole response. We suppress the fixed prompt $x$ in $p_\theta(\tau)$ and $q(\tau)$. The probability of a complete response is the product of its conditional token probabilities. Taking the logarithm turns this product into a sum; differentiating the log-probability therefore gives the sum of the token scores. Applying IS to the entire response gives the following identity (SC paper, Section 2.2):
\[\nabla_\theta J(\theta) =\mathbb E_{\tau\sim q}\!\left[ \frac{p_\theta(\tau)}{q(\tau)}R(\tau) \sum_{t=1}^{T}\nabla_\theta\log p_\theta(y_t\mid h_t) \right].\] \[\frac{p_\theta(\tau)}{q(\tau)} =\prod_{t=1}^{T}\frac{p_\theta(y_t\mid h_t)}{q(y_t\mid h_t)}.\]Each factor uses the probability of the observed token at its own prefix; the product gives the probability ratio for the full response. Include any sampled end-of-response token in this factorization. The identity assumes a parameter-independent reward, the support and finite-expectation conditions from Section 3.1 applied to responses, and probabilities matching the actual sampling procedure. A single token ratio multiplying the terminal reward generally does not equal this full-response correction. Computing ratios in log space improves numerical evaluation but does not remove sample-to-sample variation in their products.
4.2 Group Reward Centering and Conditional Score Centering
4.2 组内奖励中心化与条件 score 中心化
The SC paper, Section 4, distinguishes prompt-level reward centering from prefix-level score centering. Suppose a group contains $G$ responses to the same prompt, indexed by $i=1,\ldots,G$. Let $R_i$ be response $i$’s reward. Define the group’s mean reward $\bar R$ and centered reward $A_i$ by
\[\bar R=\frac1G\sum_{i=1}^{G}R_i,\qquad A_i=R_i-\bar R.\]Here $A_i$ is the reward-baseline difference, often called an advantage, used to weight response $i$’s token scores. By construction, $\sum_i A_i=0$. If all responses share the same prefix $h$ and the same policies, their exact score correction is the same vector $\bar s(h)$ from Section 4.1. Its equally weighted group contribution cancels:
\[\sum_{i=1}^{G}A_i\bar s(h) =\bar s(h)\sum_{i=1}^{G}A_i=0.\]This applies at the shared initial prefix. Later, let $h_{i,t}$ denote response $i$’s prefix before token $t$. These prefixes and their corrections $\bar s(h_{i,t})$ can differ, so the correction cannot in general be factored out of the sum. Group reward centering therefore does not generally make prefix-conditional score centering redundant.
4.3 Implementing SC with a Surrogate Loss
4.3 用 surrogate loss 实现 SC
The scalar-loss formulation follows the SC paper, Appendix A.3. A surrogate loss is a scalar expression constructed to produce a specified update by differentiation. Fix a prefix $h$, and abbreviate $p_\theta(a\mid h)$ and $q(a\mid h)$ as $p_\theta(a)$ and $q(a)$ in this subsection. Let $y$ be the sampled next token, and $R$ the reward of the completed response containing it.
Define $\operatorname{sg}$, or stop-gradient, to return its input value but treat it as constant during differentiation. A local SC loss is
\[L_{\rm SC}=-\operatorname{sg}(R) \left[\log p_\theta(y)-\sum_a\operatorname{sg}(q(a))\log p_\theta(a)\right],\]where the sum runs over the vocabulary. One backward pass computes its parameter gradient. The minus sign makes gradient descent on this loss implement the SC ascent direction:
\[-\nabla_\theta L_{\rm SC} =R\bigl(s_p(y,h)-\bar s(h)\bigr).\]Thus the vocabulary sum can be differentiated as one scalar expression; no separate parameter-gradient vector must be stored for each action. For an advantage-weighted update, replace $R$ by a specified coefficient such as $A_i=R_i-\bar R$ from Section 4.2 and stop its gradient. For weighted SC from Section 3.2, use detached coefficients $u(y)$ in the sampled term and $q(a)u(a)$ in the correction term. “Detached” means treated as constant by stop-gradient. Rewards and weights are coefficients in the specified update: differentiating these coefficients as well would apply the product rule and introduce terms absent from that update.
This loss implements a local correction at the fixed prefix. The distinction from a complete-response gradient remains the one described in Section 4.1.
Implementation details: annotated code
Implementation details: a top-k approximation
The official implementation stores only the $k$ highest-probability actions under the sampling policy, where $k$ is the number of recorded actions. Work at one fixed prefix and omit it from the notation as in Section 4.3. Let $H$ be the recorded action set. The omitted, or tail, probability masses are
\[P_{\mathrm{tail}}=1-\sum_{a\in H}p_\theta(a),\qquad Q_{\mathrm{tail}}=1-\sum_{a\in H}q(a).\]The subscript “tail” refers to actions outside $H$, not response length. When $P_{\mathrm{tail}}>0$, approximate the unrecorded sampling probabilities in proportion to the current policy:
\[\widehat q(a)= \begin{cases} q(a),&a\in H,\\ \dfrac{Q_{\mathrm{tail}}}{P_{\mathrm{tail}}}p_\theta(a),&a\notin H. \end{cases}\]The factor $Q_{\mathrm{tail}}/P_{\mathrm{tail}}$ makes the approximated tail probabilities sum to the recorded tail mass $Q_{\mathrm{tail}}$. To compute the resulting score mean without enumerating tail actions, use the current policy’s zero mean score:
\[\sum_{a\notin H}p_\theta(a)s_p(a,h) =-\sum_{a\in H}p_\theta(a)s_p(a,h).\]The head and tail together sum to zero, so the unknown tail contribution can be replaced by the negative head contribution. Combining it with the recorded head gives the approximate score mean $\widehat{\bar s}$:
\[\widehat{\bar s}=\sum_{a\in H} \left[q(a)-\frac{Q_{\mathrm{tail}}}{P_{\mathrm{tail}}}p_\theta(a)\right]s_p(a,h).\]Here the hat denotes an approximation, rather than a random single-sample update. Subtracting this mean centers the score under $\widehat q$, but not necessarily under the actual $q$. The construction retains the omitted probability mass; renormalizing only the recorded actions defines a different distribution. A zero or numerically negligible $P_{\mathrm{tail}}$ requires explicit handling. In a log-probability surrogate, stop gradients through the bracketed coefficients, as in Section 4.3, to avoid additional derivative terms.
5. Two Interpretations of the Centered Update
5. 对中心化更新的两种解释
5.1 Nan Jiang: the Linear-Softmax Case
5.1 Nan Jiang：linear softmax 的情形
Nan Jiang’s comment considers a linear-softmax policy. To spell out the observation, fix a feature vector $x$ and let $p=\operatorname{softmax}(Wx)$, where $W$ is the trainable weight matrix and $p$ is the action-probability vector. Let $e_a$ be the one-hot vector for action $a$. The score with respect to $W$ and its exact centered form are
\[\begin{aligned} s_W(a)&=(e_a-p)x^\top,\\ s_W(a)-\mathbb E_q[s_W]&=(e_a-q)x^\top. \end{aligned}\]The explicit dependence on $p$ cancels. If the sampler is $q=\operatorname{softmax}(Ux)$ with the same fixed features and parameter coordinates, this equals its score at weights $U$. The observation shows why SC need not recover the trainer’s target gradient. It does not mean SC leaves the update unchanged or has no effect: centering still removes the constant-reward drift.
5.2 Yingru Li: The Trainer-Tangent Policy Family
5.2 Yingru Li：从 Trainer-Tangent Policy Family 理解 SC
The construction in Yingru Li’s comment gives another way to understand SC: start from the sampler’s probabilities, apply the trainer’s relative probability changes, and normalize. Centering then follows from differentiating this normalized distribution. We will first define the distribution exactly, before making any approximation.
What changes, and what stays fixed? Fix a history or prefix $h$. The current trainer is $p_\theta(a\mid h)$, and the sampler is $q(a\mid h)$. Hold both $q$ and the reference parameter vector $\theta$ fixed. Let $\delta$ be a candidate parameter increment, so $p_{\theta+\delta}$ describes how the trainer would change. $\delta$ has the same coordinates as $\theta$; $\delta=0$ means no increment, not that the model’s parameters are zero. For simplicity, assume finitely many actions, positive probabilities, and a trainer policy differentiable near $\theta$.
For each candidate $\delta$, we will construct a new probability distribution $\nu_\delta$. The collection of distributions indexed by $\delta$ is a policy family. It is an auxiliary mathematical construction; we have not yet updated either actual model.
Apply relative changes, then normalize. The ratio
\[\frac{p_{\theta+\delta}(a\mid h)}{p_\theta(a\mid h)}\]measures how much the trainer multiplies the probability of action $a$. A ratio of $1.5$ means increasing that probability by 50%; a ratio of $0.5$ means halving it. Apply that same multiplier to the sampler’s probability, giving $q(a\mid h)p_{\theta+\delta}(a\mid h)/p_\theta(a\mid h)$. These new numbers are nonnegative, but their sum need not be one.
Define their sum as
\[\boxed{ Z_\delta(h)=\sum_u q(u\mid h) \frac{p_{\theta+\delta}(u\mid h)}{p_\theta(u\mid h)}. }\]Here $u$ is just an index running over all possible actions. $Z_\delta(h)$ is one number for this prefix and increment, shared by every action. It is called a normalization constant because dividing each weight by this total makes the probabilities sum to one:
\[\boxed{ \nu_\delta(a\mid h)= \frac{q(a\mid h)\,\dfrac{p_{\theta+\delta}(a\mid h)}{p_\theta(a\mid h)}} {Z_\delta(h)}. }\]“Constant” means common across actions, not constant as $\delta$ changes. The sum of the numerators is exactly $Z_\delta(h)$, so $\sum_a\nu_\delta(a\mid h)=Z_\delta(h)/Z_\delta(h)=1$.
Why is $\nu_0=q$? Substitute $\delta=0$ into the exact definition. For every action,
\[\frac{p_{\theta+0}(a\mid h)}{p_\theta(a\mid h)} =\frac{p_\theta(a\mid h)}{p_\theta(a\mid h)}=1.\]Therefore
\[Z_0(h)=\sum_u q(u\mid h)\times1=1, \qquad \boxed{\nu_0(a\mid h)=\frac{q(a\mid h)\times1}{1}=q(a\mid h).}\]No relative change has been applied, so we recover the sampler. There is no requirement that $q=p_\theta$. The two roles are different: the base distribution is $q$; the relative-change rule comes from the current trainer.
Differentiate the normalized family. Keep the history fixed and omit $h$ below. Write $\psi_\theta(a)=\nabla_\theta\log p_\theta(a)$, the same trainer score previously denoted $s_p(a)$. Taking the logarithm of the exact definition gives
\[\log\nu_\delta(a) =\log q(a)+\log p_{\theta+\delta}(a) -\log p_\theta(a)-\log Z_\delta.\]We differentiate with respect to the increment $\delta$. The terms $\log q(a)$ and $\log p_\theta(a)$ are fixed, so their derivatives are zero. The numerator term varies through $\theta+\delta$, giving $\psi_\theta(a)$ when evaluated at $\delta=0$.
The remaining term is the normalization. Differentiate its finite sum:
\[\begin{aligned} \left.\nabla_\delta Z_\delta\right|_{\delta=0} &=\sum_u\frac{q(u)}{p_\theta(u)} \left.\nabla_\delta p_{\theta+\delta}(u)\right|_{\delta=0}\\ &=\sum_u\frac{q(u)}{p_\theta(u)}\nabla_\theta p_\theta(u)\\ &=\sum_u q(u)\psi_\theta(u) =\mathbb E_{u\sim q}[\psi_\theta(u)]. \end{aligned}\]The third line uses $\nabla_\theta p_\theta=p_\theta\nabla_\theta\log p_\theta$.
Why does a derivative with respect to the increment become a parameter gradient? (click to expand)
The differentiation variable has not been changed midway. The reference parameter $\theta$ stays fixed while the increment $\delta$ varies. To distinguish a variable from the point where a derivative is evaluated, introduce a separate parameter variable $w$ and, for one fixed action $u$, define
\[f(w)=\log p_w(u),\qquad w=\theta+\delta.\]Our expression is now $f(\theta+\delta)$. Changing one coordinate $\delta_i$ changes $w_i=\theta_i+\delta_i$ by exactly the same amount, leaving every other coordinate unchanged. The chain rule therefore gives
\[\nabla_\delta f(\theta+\delta) =\left.\nabla_w f(w)\right|_{w=\theta+\delta}.\]The left side measures change with respect to the increment; the right side measures change with respect to the actual parameter. They agree because adding a fixed $\theta$ only shifts the origin of the parameter coordinates.
Differentiate first, then evaluate at $\delta=0$. The vertical bar denotes this second operation. At zero increment, the actual parameter is $w=\theta$, so
\[\boxed{ \left.\nabla_\delta\log p_{\theta+\delta}(u)\right|_{\delta=0} =\left.\nabla_w\log p_w(u)\right|_{w=\theta} =\psi_\theta(u). }\]The middle expression is commonly abbreviated as $\nabla_\theta\log p_\theta(u)$: differentiate the policy’s log-probability with respect to its parameter, then evaluate at the current parameter value. This does not mean differentiating the entire $Z_\delta$ expression with respect to its reference $\theta$. In our calculation, the denominator $p_\theta(u)$ and $q(u)$ remain fixed.
A scalar example of the chain rule, unrelated to probabilities, makes the order explicit. With reference value $\theta=3$ and $f(w)=w^2$,
\[\left.\frac{d}{d\delta}(3+\delta)^2\right|_{\delta=0} =\left.2(3+\delta)\right|_{\delta=0} =6 =\left.\frac{d}{dw}w^2\right|_{w=3}.\]Substituting zero before differentiation would leave only the constant $9$; differentiating that constant gives zero, not the required slope $6$. It discards how the original function changes near that point. A function’s value at a point is not its derivative there.
Return to the normalization factor. Keeping $\delta$ unevaluated during differentiation and using $\nabla p=p\nabla\log p$ gives
\[\nabla_\delta Z_\delta =\sum_u q(u)\frac{p_{\theta+\delta}(u)}{p_\theta(u)} \nabla_\delta\log p_{\theta+\delta}(u).\]The probability ratio must still be present here. Only when we now evaluate at $\delta=0$ does it become one. The remaining derivative is the parameter gradient evaluated at $w=\theta$:
\[\left.\nabla_\delta Z_\delta\right|_{\delta=0} =\mathbb E_{u\sim q}\!\left[ \left.\nabla_w\log p_w(u)\right|_{w=\theta} \right] =\mathbb E_{u\sim q}[\psi_\theta(u)].\]This expectation is the desired result: the trainer score averaged under the fixed sampler $q$. It need not be zero, because the averaging distribution need not be $p_\theta$.
Since the derivative of a logarithm is the derivative of its argument divided by that argument, and $Z_0=1$,
\[\left.\nabla_\delta\log Z_\delta\right|_{\delta=0} =\frac{1}{Z_0} \left.\nabla_\delta Z_\delta\right|_{\delta=0} =\mathbb E_q[\psi_\theta].\]Knowing that $Z_0=1$ tells us its value at one point, not that its derivative is zero. Substituting the derivative into the log-probability expression yields
\[\boxed{ \left.\nabla_\delta\log\nu_\delta(a)\right|_{\delta=0} =\psi_\theta(a)-\mathbb E_q[\psi_\theta] =\widetilde\psi_{\theta,q}(a). }\]The vertical bar means “evaluate the derivative at $\delta=0$.” This is exactly the centered score used by SC. The subtracted mean comes from differentiating the normalization term, not from a separate correction added by hand. This score identity is exact.
What the first-order expansion reveals. We can also see why normalization requires centering by expanding small changes. Let $|\delta|$ measure the size of the parameter increment. The notation $o(|\delta|)$ denotes a remainder whose magnitude divided by $|\delta|$ tends to zero as the increment tends to zero. Differentiability gives
\[\frac{p_{\theta+\delta}(a)}{p_\theta(a)} =1+\delta^\top\psi_\theta(a)+o(\|\delta\|).\]The dot product $\delta^\top\psi_\theta(a)$ is the first-order relative change associated with this parameter move. Summing the reweighted sampler probabilities gives
\[Z_\delta =1+\delta^\top\mathbb E_q[\psi_\theta]+o(\|\delta\|).\]Using the first-order identity $1/(1+x)=1-x+o(\lvert x\rvert)$ for the normalization factor, we obtain
\[\boxed{ \nu_\delta(a) =q(a)\left[ 1+\delta^\top\bigl(\psi_\theta(a)-\mathbb E_q[\psi_\theta]\bigr) \right]+o(\|\delta\|). }\]Before normalization, the first-order changes sum to $\delta^\top\mathbb E_q[\psi_\theta]$, which need not be zero. Normalization removes this shared relative component. The centered changes sum to zero because
\[\sum_a q(a)\bigl(\psi_\theta(a)-\mathbb E_q[\psi_\theta]\bigr)=0.\]Thus probability can move between actions without creating or destroying total probability mass. The first-order formula approximates a nearby distribution; the exact ratio-and-normalization definition guarantees valid probabilities.
The name trainer-tangent policy family refers to this local construction: the family starts at the sampler, while the trainer supplies its relative-change rule. For a fixed reward $R(a)$ at this fixed history,
\[\left.\nabla_\delta\mathbb E_{a\sim\nu_\delta}[R(a)]\right|_{\delta=0} =\mathbb E_{a\sim q} \left[R(a)\widetilde\psi_{\theta,q}(a)\right].\]The right-hand side is the expected exact-SC update. This identifies the local objective of the auxiliary family; it does not turn it into the current trainer’s expected-reward gradient. The essential explanation is: sampler as the starting point, trainer-relative changes as the rule, and normalization as the source of score centering.