Score Centering: Drift, Policy Gradients, and the Off-Policy Debate
Imagine rewarding every generated answer with exactly one point. There is no better answer to discover. Should reinforcement learning still systematically change the model? If the model generating answers and the model computing gradients disagree, the answer can be yes.
This is the entry point to Score Centering Stabilizes Off-policy Reinforcement Learning, by Martin Marek and Max Ryabinin. We will derive the mechanism ourselves, first with two possible answers, then with an entire generated sequence. The crucial distinction is between removing an unwanted average push, recovering a particular target gradient, and making training work better. They are three different claims.
The discussion starts from Nan Jiang’s post. The complete reply chain was not accessible during preparation; the later exchange is reconstructed from the supplied discussion, not presented as a verbatim transcript or verified chronology. Mathematical counterexamples and judgments below are this article’s analysis.
1. One Training Loop, Two Distributions
1. 一个训练循环，为什么有两个分布？
1.1 Who Generates, and Who Learns?
1.1 谁生成数据，谁根据数据学习？
A policy is a rule that assigns probabilities to possible actions. For an LM, the action is usually the next token. At a fixed prefix $h$, the training model assigns probability $p_\theta(a\mid h)$ to token $a$. The vector $\theta$ contains the parameters that the optimizer will update.
But the probabilities that actually produce the token may be $q(a\mid h)$. A faster rollout engine can use quantized weights, different numerical kernels, or an older checkpoint. Here rollout means generating a response or trajectory to be scored. Training–inference mismatch (TIM) means the sampling probabilities and training probabilities disagree; it is not merely that two random draws happened to differ.
We hold the collected sampling distribution fixed when computing one training update. Even if $q$ came from $\theta$, a saved rollout does not change retroactively when the trainer changes. This bookkeeping does not claim that $q$ is unrelated to $p$ across training iterations.
1.2 What Does “The Correct Gradient” Mean?
1.2 先确定：“正确梯度”是哪个目标的梯度？
Start with one decision and two outcomes, $y=1$ and $y=0$. Write $p=p_\theta(1)$ and $q=q(1)$; the other probabilities are $1-p$ and $1-q$. This scalar shorthand is only for the binary example. Later, $p_\theta$ and $q$ denote full distributions.
Let $r_1$ and $r_0$ be the rewards of the two outcomes, fixed independently of $\theta$. The trainer’s objective is
\[J(\theta)=\mathbb E_{y\sim p_\theta}[R(y)]=p\,r_1+(1-p)r_0.\]The expectation means “average using the frequencies the current training policy would produce.” We want $\nabla_\theta J$, not the gradient of average reward under an arbitrary other policy.
We use gradient ascent, $\theta_{\rm new}=\theta+\eta\widehat g$, because larger reward is better. Here $\widehat g$ is a sampled update estimate and $\eta>0$ is the learning rate. A loss-minimizing implementation uses the negative objective. Expectation calculations below concern $\mathbb E[\widehat g]$, before multiplication by the learning rate or processing by Adam. Gradient basics are developed in Basics of Optimizers, §1.
For now there is no KL penalty, PPO clipping, learned reward dependence on $\theta$, or group normalization. These alter the objective or estimator and must not be silently included in a claim about the gradient of $J$.
2. From a Probability to a Policy Gradient
2. 从概率开始，一步步得到 Policy Gradient
2.1 Why Is the Score y − p?
2.1 Score 是什么，为什么等于 y − p？
A parameter must be free to move without making probabilities negative or larger than one. Use a real-valued logit $\theta$, converted to a probability by the sigmoid:
\[p=\sigma(\theta)=\frac1{1+e^{-\theta}},\qquad \frac{dp}{d\theta}=p(1-p).\]A logit is not a log-probability. The log-probability of the observed binary outcome is
\[\log p_\theta(y)=y\log p+(1-y)\log(1-p).\]The score is the derivative of that log-probability with respect to the trainable parameter. It is not the reward assigned by a judge. With natural logarithms,
\[\begin{aligned} s_p(y) &=\frac{d}{d\theta}\log p_\theta(y)\\ &=y\frac{p(1-p)}p-(1-y)\frac{p(1-p)}{1-p}\\ &=y(1-p)-(1-y)p\\ &=\boxed{y-p}. \end{aligned}\]If $y=1$, the score is $1-p>0$: increasing the logit would make that sampled answer more likely. If $y=0$, the score is $-p<0$: decreasing it would make zero more likely. In a full network, this derivative becomes a vector over parameters; it supplies a local direction, not a command to move a fixed distance.
Why does differentiating the sigmoid give p(1 − p)?
Differentiating $(1+e^{-\theta})^{-1}$ gives $e^{-\theta}/(1+e^{-\theta})^2$. Meanwhile $p=1/(1+e^{-\theta})$ and $1-p=e^{-\theta}/(1+e^{-\theta})$. Multiplying them gives exactly the derivative. The two factors also explain why changing a logit has little effect on probability near either endpoint.
2.2 Derive the Target Gradient in Two Ways
2.2 用两种方法算出目标梯度
First differentiate the objective directly:
\[\begin{aligned} J(\theta)&=r_0+p(r_1-r_0),\\ g_p=\frac{dJ}{d\theta}&=\boxed{p(1-p)(r_1-r_0)}. \end{aligned}\]The reward difference determines which outcome should become more likely. The factor $p(1-p)$ converts a change in logit to a change in probability. If both rewards are equal, the objective is constant and its gradient must be zero.
Second, weight each score by its reward and average under p:
\[\begin{aligned} \mathbb E_p[R\,s_p] &=p\,r_1(1-p)+(1-p)r_0(-p)\\ &=p(1-p)(r_1-r_0)=g_p. \end{aligned}\]This is the policy-gradient identity in miniature. Its general finite-action derivation is just a rearrangement:
\[\nabla_\theta\sum_a p_\theta(a)R(a) =\sum_a R(a)\nabla_\theta p_\theta(a) =\sum_a p_\theta(a)R(a)\nabla_\theta\log p_\theta(a).\]The equality uses $\nabla p=p\nabla\log p$, valid for positive probabilities. Instead of differentiating a random discrete sample, we differentiate its log-probability and use samples to estimate the sum. The distribution specifying the frequencies is therefore part of the mathematics, not an implementation detail.
2.3 Keep the Score, Change the Sampling Frequencies
2.3 Score 没算错，为什么平均更新仍然错了？
Now draw $y$ from $q$, but keep computing the trainer’s score $y-p$. The expectation becomes
\[g_{\rm raw}=\mathbb E_q[R(y-p)] =q\,r_1(1-p)-(1-q)r_0p.\]There is nothing wrong with the derivative of $\log p_\theta(y)$. The problem is that we average its possible values with $q$ rather than $p$. More data from the same sampler makes the estimate converge more accurately to $g_{\rm raw}$, not to $g_p$. This systematic difference in means is bias, as distinct from finite-sample randomness.
Set $r_1=r_0=1$. Then $g_p=0$, but
\[g_{\rm raw}=q(1-p)-(1-q)p=q-p.\]If the sampler favors outcome one more strongly than the trainer, the trainer receives a positive average push despite the reward expressing no preference. This is the constant-reward diagnostic, not yet a proof that every such mismatch causes collapse.
3. What Score Centering Removes
3. Score Centering 究竟移除了什么？
3.1 Separate the Common Push from Reward Differences
3.1 把共同推力与奖励差异分开
For the next calculation, fix one prefix and suppress $h$ in the notation. Write $s_p(a)=\nabla_\theta\log p_\theta(a)$ and $\bar s=\mathbb E_q[s_p]$. Under the trainer’s own distribution,
\[\mathbb E_p[s_p] =\sum_a p_\theta(a)\nabla_\theta\log p_\theta(a) =\nabla_\theta\sum_a p_\theta(a)=0.\]Under $q$, the same zero-mean identity need not hold. Let $\mu_R=\mathbb E_q[R]$. Split a reward into its common level and deviation: $R=\mu_R+(R-\mu_R)$. Multiplying by the score and taking expectations gives
\[\boxed{\mathbb E_q[R s_p] =\underbrace{\mu_R\bar s}_{\text{drift}} +\underbrace{\operatorname{Cov}_q(R,s_p)}_{\text{reward--score association}}}.\]Here covariance means $\mathbb E_q[(R-\mu_R)(s_p-\bar s)]$, evaluated component by component if the score is a vector. The extra $-\bar s$ does not change the answer because $\mathbb E_q[R-\mu_R]=0$.
Intuitively, drift asks only “what was the average reward?” and multiplies that number by the sampler’s average score. It does not ask which action earned more. Covariance does ask that: do above-average rewards occur together with a particular update direction? Calling it “signal” does not make it the correct signal for $p$; its average is still taken under $q$.
For our binary model, $\bar s=q-p$, $\mu_R=qr_1+(1-q)r_0$, and direct substitution gives
\[\boxed{g_{\rm raw} =\underbrace{[qr_1+(1-q)r_0](q-p)}_{\text{drift}} +\underbrace{q(1-q)(r_1-r_0)}_{\text{covariance}}}.\]3.2 Subtract the Average Score, Not the Average Reward
3.2 减掉的是平均 Score，不是平均 Reward
Score centering replaces $s_p(y)$ by
\[\widetilde s(y)=s_p(y)-\bar s,\qquad \mathbb E_q[\widetilde s]=0.\]At a fixed prefix, $\bar s$ is the same vector regardless of which next token is sampled. Different prefixes generally have different vectors. It is an expectation over possible next tokens, not a mean across the coordinates of one gradient.
The reward-weighted expectation becomes
\[g_{\rm SC}=\mathbb E_q[R(s_p-\bar s)] =\mathbb E_q[Rs_p]-\mathbb E_q[R]\bar s =\operatorname{Cov}_q(R,s_p).\]In the binary case, every sampled update simplifies:
\[\boxed{R(y-p)\ \longrightarrow\ R[(y-p)-(q-p)]=R(y-q)}.\]Consequently $g_{\rm SC}=q(1-q)(r_1-r_0)$. For constant rewards this is zero. More generally, adding the same constant $c$ to every reward leaves the expected SC update unchanged, since $\mathbb E_q[c\widetilde s]=0$. The raw update changes by $c\bar s$.
This is an expectation statement. With reward one, a single centered sample can still be nonzero, and a finite minibatch can move randomly. Zero mean does not mean zero variance or guaranteed optimization stability.
3.3 Why Might a Small Drift Matter over Training?
3.3 为什么一点 Drift 可能在训练中积累？
For fixed $q$, the mean score is
\[\bar s=\nabla_\theta\sum_a q(a)\log p_\theta(a).\]It is the direction used to fit the trainer to the sampler by maximum likelihood. With a positive mean reward, drift points toward this fitting direction; with a negative mean reward, it reverses it. This is a distribution-matching pressure, not evidence from relative action quality.
A simple constructed loop illustrates accumulation. Suppose at iteration $n$ the sampler uses logit $\theta_n+\delta$ with a persistent $\delta>0$, so $p_n=\sigma(\theta_n)$ and $q_n=\sigma(\theta_n+\delta)$. With reward always one, the raw expected update is positive at every finite $\theta_n$:
\[\theta_{n+1}=\theta_n+\eta(q_n-p_n).\]The sampler is then refreshed from the new trainer, retaining the offset. The target it pushes toward keeps moving. SC eliminates this particular expected push. This example demonstrates a mechanism, not that real quantization behaves as a fixed scalar offset, or that every mismatch produces divergent rewards.
It also explains why “SFT uses off-policy data too” is not a rebuttal. Fitting a fixed dataset is SFT’s intended objective. Here the intended objective is reward under $p$, and the effective teacher changes with the learner. Neither statement implies that SFT or a moving teacher is universally stable or unstable.
4. The Disagreement: Correcting toward What?
4. 争论的核心：究竟在校正到哪里？
4.1 The Strongest Diagnostic: p Disappears
4.1 最有力的观察：p 消失了
Return to $R(y-p)\to R(y-q)$. Hold $q$ and the observed outcome fixed, and replace the trainer by a different $p$. The centered update is unchanged, although the correct gradient $p(1-p)(r_1-r_0)$ generally changes. This is the strongest reading of Nan’s objection: the correction does not use the target-policy difference to recover the target gradient.
“Does nothing,” in this reading, does not mean the original and corrected formulas are equal. Their difference is $R(q-p)$, usually nonzero. Nor does it refer to the special case where one-step group centering already cancels a shared correction. It means “does not perform the claimed target-directed correction.” The phrase is too broad if interpreted as “removes no bias component” or “has no practical effect.”
Let $q=\sigma(\phi)$. The expected centered update is exactly $dJ_q/d\phi=q(1-q)(r_1-r_0)$ in this binary parameterization. That is a derivative with respect to a sampler logit at $q$, applied as a direction to the trainer logit. It is not the derivative of a fixed $q$ with respect to $\theta$, which would be zero.
4.2 What Does the Linear-Softmax Qualification Exclude?
4.2 为什么必须说明 Linear Softmax 的假设？
For more than two outcomes, let $x$ be a fixed feature vector and let $p=\operatorname{softmax}(Wx)$. Let $e_y$ be a vector with one at the sampled token and zeros elsewhere. Differentiating with respect to the weight matrix gives
\[s_W(y)=(e_y-p)x^\top,\qquad \mathbb E_q[s_W]=(q-p)x^\top.\]Thus the centered score is $(e_y-q)x^\top$. If the sampler is another linear softmax with the same fixed features and parameter coordinates, this is its score evaluated at its own weights. This is a precise equivalence, not a universal description of neural networks.
In a nonlinear network, let $z_\theta(h)$ be the logit vector and let $J_z=\partial z_\theta(h)/\partial\theta$ be its Jacobian: each entry records how one logit changes when one parameter changes. The chain rule gives
\[s_\theta(y)=J_z^\top(e_y-p),\qquad \widetilde s_\theta(y)=J_z^\top(e_y-q).\]The explicit probability vector $p$ cancels, but the trainer still appears through $J_z$. A quantized or stale sampler can have a different Jacobian, or no useful ordinary derivative through its discrete quantization operation. So one cannot generally call this the true sampler-parameter gradient. The linear example establishes a limitation of target correction; it does not prove that all trainer information disappears in a full LM.
4.3 Removing Drift Can Increase the Total Bias
4.3 移除 Drift，反而可能离目标更远
Use raw rewards and one decision:
\[p=0.9,\qquad q=0.8,\qquad r_1=1,\quad r_0=0.\]| Update | Expected value | Distance from the target |
|---|---|---|
| Correct gradient at $p$ | $p(1-p)=0.09$ | $0$ |
| Raw off-policy update | $q(1-p)=0.08$ | $0.01$ |
| Score-centered update | $q(1-q)=0.16$ | $0.07$ |
Here the mean reward is $0.8$, so drift is $0.8(0.8-0.9)=-0.08$. Covariance is $0.16$, while the target is $0.09$. The covariance’s error is therefore $+0.07$. Before correction, the two errors largely cancel:
\[g_{\rm raw}-g_p=-0.08+0.07=-0.01.\]SC deletes the $-0.08$ term and leaves the $+0.07$ error. Deleting one component of a sum does not guarantee reducing the magnitude of the sum.
This counterexample concerns expected gradients, not one lucky minibatch. It disproves a universal closeness guarantee, but does not predict which estimator has better finite-sample mean squared error or better long-run reward. Those also depend on variance, learning rate, optimizer, and changing policies.
There is also a useful favorable fact in this particular binary setting: for interior $p,q$, both $g_p$ and $g_{\rm SC}$ have the sign of $r_1-r_0$. Their magnitudes can differ, but SC is still a local ascent direction when that difference is nonzero. The counterexample says “not closer,” not “necessarily the wrong direction.” This sign guarantee does not automatically survive shared parameters across many states or actions.
4.4 Two Different Meanings of Correction
4.4 两种“校正”，可以同时得出不同结论
The exchange beginning at Nan’s post, as reconstructed from the supplied follow-up, admits a clear distinction. Nan’s target-gradient objection survives the calculation: centering does not generally recover $g_p$, nor guarantee moving closer to it. You Jiacheng’s drift-removal defense survives too: with exact centering, the defined drift really is removed. It is not an unchanged estimator.
The defense that $q$ is derived from $p$ supplies an engineering reason the update might be useful. A nearby or slightly older policy’s update may remain informative. But provenance is not an equality of gradients. For our binary example,
\[g_{\rm SC}-g_p=(q-p)(1-p-q)(r_1-r_0).\]Hence $\lvert g_{\rm SC}-g_p\rvert\leq\lvert q-p\rvert\,\lvert r_1-r_0\rvert$. Closeness of $q$ to $p$ helps in this toy model. It does not prove SC is closer than the raw estimator, and “derived from” alone supplies no numerical closeness bound.
Our assessment is therefore specific: Nan has the stronger argument about recovering the current trainer’s gradient; You correctly defends removing a particular drift term. Comparing training stability or final reward against IS-based methods is meaningful. Interpreting superior reward as proof of more accurate target-gradient recovery is not.
5. How Importance Sampling Restores the Target Average
5. Importance Sampling 怎样恢复目标梯度？
5.1 Correct Frequencies, Keep the Trainer's Score
5.1 调整出现频率，保留 Trainer 的 Score
If outcome one should appear 90% of the time but appears only 80%, each observed one can count as $0.9/0.8=1.125$ target samples. Zeros appear 20% rather than 10%, so each zero counts as $0.1/0.2=0.5$ samples. We do not need to change the sampled token; we change how much it contributes.
In general, define the importance weight
\[w(y)= \begin{cases} p/q,&y=1,\\ (1-p)/(1-q),&y=0. \end{cases}\]Use $\widehat g_{\rm IS}=w(y)R(y)(y-p)$, retaining the trainer’s score:
\[\begin{aligned} \mathbb E_q[\widehat g_{\rm IS}] &=q\frac pq r_1(1-p) +(1-q)\frac{1-p}{1-q}r_0(-p)\\ &=p\,r_1(1-p)-(1-p)r_0p\\ &=\boxed{p(1-p)(r_1-r_0)}. \end{aligned}\]The sample frequencies $q$ cancel the denominators. For the counterexample, this is $0.8\times1.125\times0.1=0.09$. SC changed each sample’s score; IS changes its contribution to the average.
For any integrable function $f$, the underlying identity is $\sum_a q(a)[p(a)/q(a)]f(a)=\sum_a p(a)f(a)$. It requires support coverage: whenever $p(a)>0$, we need $q(a)>0$. An action the sampler never produces cannot be recovered merely by assigning weights to observed actions.
5.2 Unbiased Does Not Mean Easy to Optimize
5.2 无偏，不代表容易训练
Suppose $p=0.5$, $q=0.001$, $r_1=1$, and $r_0=0$. The correct logit gradient is $0.25$. The IS estimator is 250 on the rare draw $y=1$, and zero otherwise:
\[\mathbb E[\widehat g_{\rm IS}]=0.001\times250=0.25,\qquad \operatorname{Var}(\widehat g_{\rm IS})=0.001\times250^2-0.25^2=62.4375.\]An expectation can be correct while most batches miss the informative action and a few batches receive an enormous update. Increasing batch size helps if variance is finite, but costs computation and does not solve missing support.
Truncated IS replaces $w$ by, for example, $\min(w,c)$. Masked IS sets weights outside an accepted range to zero. Both suppress extreme contributions but generally break the exact cancellation above. Self-normalizing weights by their batch sum is also generally biased at finite sample size. These may be worthwhile tradeoffs; they are not the exact identity.
Nor is there a theorem that IS always has higher variance than every unweighted estimator, or SC always lowers variance. With constant reward, subtracting a deterministic score mean changes the mean but leaves the score variance unchanged. A claim about stability needs more than the word “centering.”
For independent scalar estimates averaged over $N$ samples, the mean squared error is
\[\mathbb E[(\widehat g_N-g_p)^2] =(\mathbb E[\widehat g_N]-g_p)^2+\frac{\operatorname{Var}(\widehat g_1)}N.\]This separates squared bias from variance. Long-run learning quality is a further question: the objective and both policies evolve after each step.
6. From One Decision to an Entire LM Response
6. 从一次选择到完整 LM 回答
6.1 Why the Exact Ratio Is a Product
6.1 为什么精确的权重是整条轨迹的概率比？
Fix a prompt $x$. Let $\tau=(y_1,\ldots,y_T)$ be a generated response, including a stopping token when relevant. The prefix before token $t$ is $h_t=(x,y_{<t})$. A trajectory is simply all these decisions together. Autoregressive generation means
\[p_\theta(\tau\mid x)=\prod_{t=1}^T p_\theta(y_t\mid h_t), \qquad q(\tau\mid x)=\prod_{t=1}^T q(y_t\mid h_t).\]Taking a logarithm turns the product into a sum, so the response score is
\[\nabla_\theta\log p_\theta(\tau\mid x) =\sum_{t=1}^T s_\theta(y_t,h_t).\]For a terminal reward $R(\tau)$ independent of $\theta$, the exact objective gradient is
\[\boxed{\nabla_\theta J =\mathbb E_{\tau\sim p_\theta} \left[R(\tau)\sum_t s_\theta(y_t,h_t)\right]}.\]Changing the sampling measure to $q$ requires the probability ratio of that same random object, the entire trajectory:
\[\boxed{w(\tau)=\frac{p_\theta(\tau\mid x)}{q(\tau\mid x)} =\prod_t\frac{p_\theta(y_t\mid h_t)}{q(y_t\mid h_t)}}.\]Then $\mathbb E_q[w(\tau)R(\tau)\sum_t s_\theta]=\nabla_\theta J$, assuming support coverage and integrability. Variable response lengths must be accounted for through stopping probabilities, not silently dropped. With environment transitions shared by the two policies, their factors cancel in the trajectory ratio. Different prompt distributions or environments require additional accounting.
Each denominator must be the probability actually used when that token was sampled, including any temperature or sampling filter. Recomputing it later under an updated sampler does not describe the original data. Top-$k$ or top-$p$ sampling can remove support altogether, making exact recovery of an unfiltered target impossible from those samples alone.
6.2 Why One Token's Ratio Is Usually Insufficient
6.2 为什么只乘一个 Token 的概率比通常不够？
At a fixed prefix, a ratio $p(a\mid h)/q(a\mid h)$ corrects the next-action frequencies. But a response-level gradient also depends on how often we reach that prefix and what happens afterward.
A two-token example isolates the first problem. Only the first token “A” allows a reward; otherwise reward is zero. Under $p$, A appears with probability $0.9$; under $q$, with probability $0.5$. At prefix A, the second token is binary with the same probability $0.5$ under both policies, and reward equals that second binary outcome. Let $\theta$ affect only this second decision.
The correct derivative is $0.9\times0.5(1-0.5)=0.225$. The current-token ratio at the second step is exactly one, yet the corresponding estimate under $q$ has mean $0.5\times0.25=0.125$. It is wrong because A appears too rarely. The first-step ratio $0.9/0.5$ restores the missing frequency.
There is a second issue for terminal rewards. Define $Q^p(h,a)$ as the expected eventual reward after choosing $a$ at $h$ and continuing with $p$. Rollouts continued under $q$ generally estimate $Q^q(h,a)$ instead. Correcting only the action at $h$ fixes neither the preceding prefix distribution nor this future continuation distribution.
More formally, if $d_t^p(h)$ is the probability of prefix $h$ at step $t$ under $p$, the exact gradient is
\[\nabla_\theta J =\sum_t\mathbb E_{h\sim d_t^p,\ a\sim p(\cdot\mid h)} \left[Q^p(h,a)s_\theta(a,h)\right].\]Exact alternative estimators can use appropriate prefix ratios, per-decision weighting, or a correct target-policy value function. The point is not that full-trajectory weighting is the only possibility. It is that a local token ratio alone is not an exact full-trajectory correction in general.
For long responses, products of ratios can become extremely uneven; computing the product via sums of log-ratios avoids numerical underflow but does not fix statistical variance. Clipping, filtering, and token-local surrogates address practical difficulties at the price of changing the estimator.
6.3 Why GRPO's Group Centering Does Not Settle the Problem
6.3 GRPO 已经减了平均奖励，为什么还会有 Drift？
For $G$ responses to one prompt, simple group centering forms $A_i=R_i-\bar R$, where $\bar R=G^{-1}\sum_iR_i$. The advantage $A_i$ says whether response $i$ scored above the group average. Thus $\sum_i A_i=0$. GRPO variants may additionally divide by a group standard deviation; we isolate mean subtraction first.
At the shared first prefix, a correction vector $c$ really does cancel from this finite group:
\[\sum_i A_i(s_i-c)=\sum_i A_i s_i-c\underbrace{\sum_i A_i}_{0}.\]This is a valid special case, not an explanation of the general linear-softmax critique. After responses branch, their prefixes differ, so their correction vectors $c(h_{i,t})$ differ. Then $\sum_i A_i c(h_{i,t})$ need not vanish.
Here is a concrete conditional example. Half the sampled responses enter a “good” prefix whose eventual reward is always one; half enter a “bad” prefix whose reward is always zero. Subtract the exact prompt-level mean, $0.5$. At the good prefix, expected advantage is $+0.5$; at the bad prefix it is $-0.5$. Suppose an irrelevant next token has $q=0.8$, $p=0.6$. Its average score is $0.2$, although its choice does not affect reward. Raw local pushes are therefore $+0.1$ and $-0.1$ at the two prefixes; local SC removes both. If the prefixes use different parameter directions, even their global vector sum need not cancel.
The exact prefix baseline $b(h)=\mathbb E_q[R\mid h]$ would instead yield
\[\mathbb E_q[(R-b(h))s_p\mid h] =\operatorname{Cov}_q(R,s_p\mid h).\]This equals the expected SC update at that prefix. But it requires the correct prefix-specific expected return, not just a group mean at the initial prompt. It still gives a $q$-covariance, not a recovered $p$-gradient.
Finite groups: subtracting a sample mean is not quite an independent baseline
For independent, identically distributed one-step pairs $(R_i,s_i)$, including the current reward in $\bar R$ gives
\[\mathbb E\!\left[\frac1G\sum_i(R_i-\bar R)s_i\right] =\left(1-\frac1G\right)\operatorname{Cov}(R,s).\]To see this, $\mathbb E[\bar R s_i]=G^{-1}\mathbb E[R_i s_i]+(G-1)G^{-1}\mathbb E[R]\mathbb E[s]$. Subtracting it leaves the factor above. A leave-one-out mean uses only other independent rewards and removes this factor. Dividing by a random group standard deviation is another change; neither calculation by itself describes every GRPO implementation.
7. Turning the Update into a Loss
7. 怎样把更新写成真正可求导的 Loss？
7.1 One Scalar Loss, No Per-Token Parameter Jacobians
7.1 不必给词表中每个 Token 单独存一条梯度
The formula $\bar s=\sum_a q(a)\nabla_\theta\log p_\theta(a)$ may appear to require one enormous parameter-gradient vector for every vocabulary entry. Linearity of differentiation avoids that. At a fixed prefix, define a scalar surrogate loss:
\[L_{\rm SC} =-\operatorname{sg}(R) \left[\log p_\theta(y)-\sum_a\operatorname{sg}(q(a))\log p_\theta(a)\right].\]Here $\operatorname{sg}$ means stop-gradient: keep the numerical value but do not differentiate through its calculation. Differentiating the scalar produces
\[-\nabla_\theta L_{\rm SC}=R(s_p(y)-\bar s).\]Thus gradient descent on this loss implements the desired ascent update in one backward pass. “Surrogate” matters: its gradient specifies a local update; the scalar is not necessarily the original expected-reward objective or an unbiased estimate of its value.
Rewards or advantages are treated as fixed multipliers. Logged sampler probabilities are fixed too. If a coefficient is calculated using current trainer probabilities, it may still need to be detached to implement a specified update. Numerical dependence and differentiation paths are different questions.
For IS there are two valid scalar constructions when $q$ is fixed. Differentiating $-R\,p_\theta(y)/q(y)$ gives $-Rw s_p$. Alternatively, differentiating $-\operatorname{sg}(Rw)\log p_\theta(y)$ gives the same gradient at the evaluation point. But leaving $w$ differentiable in the latter expression adds an unwanted product-rule term:
\[\nabla_\theta[w\log p_\theta(y)] =w(1+\log p_\theta(y))s_p(y).\]The snippet uses the detached-weight surrogate convention consistently.
7.2 Combining SC with IS: Center the Weighted Score
7.2 SC 与 IS 怎样组合？要 Center 加权后的 Score
Let $u(a)$ be an action weight: one for plain SC, the exact ratio for IS, or a clipped/masked ratio. Start from the weighted score $u(a)s_p(a)$, and compute its mean
\[c_u=\mathbb E_q[u(a)s_p(a)].\]The centered weighted update is
\[\boxed{\widehat g_{u+\rm SC} =R(y)\left[u(y)s_p(y)-c_u\right]}.\]Its conditional expectation is $\operatorname{Cov}_q(R,u s_p)$, so constant reward again gives zero. A full-vocabulary surrogate is
\[L_{u+\rm SC} =-\operatorname{sg}(R)\left[ \operatorname{sg}(u(y))\log p_\theta(y) -\sum_a\operatorname{sg}(q(a)u(a))\log p_\theta(a) \right].\]For exact, untruncated local IS, $u=p/q$, and $c_u=\sum_a p(a)s_p(a)=0$. There is no remaining local drift for SC to remove. Clipped or masked weights generally have $c_u\neq0$, so an additive correction can still change their update. This local identity does not remove the trajectory issues in §6.
It would be a different operation to multiply an already centered score by IS:
\[\mathbb E_q[wR(s_p-\bar s)] =g_p-\bar s\,\mathbb E_p[R]\]in the one-step setting. The extra term generally destroys exact recovery. “Combine SC and IS” therefore needs an explicit formula; the order is not interchangeable.
7.3 What Changes When Only Top-k Probabilities Are Logged?
7.3 只记录 Top-k 概率时，哪里变成了近似？
Logging every vocabulary probability at every generated position is expensive. The supplied discussion describes the paper’s top-$k$ approach; the official implementation specifies $k=128$ and a tail proportional to trainer probabilities. Here is why that approximation yields a short correction.
Let $H$ be the saved token set, $P_T=1-\sum_{a\in H}p(a)$, $Q_T=1-\sum_{a\in H}q(a)$, and $\rho=Q_T/P_T$, assuming $P_T>0$. Define
\[\widehat q(a)= \begin{cases} q(a),&a\in H,\\ \rho p(a),&a\notin H. \end{cases}\]The tail has the right total mass but assumes the trainer has the right relative probabilities within it. Using $\sum_a p(a)s_p(a)=0$,
\[\begin{aligned} \widehat{\bar s} &=\sum_{a\in H}q(a)s_p(a)+\rho\sum_{a\notin H}p(a)s_p(a)\\ &=\sum_{a\in H}[q(a)-\rho p(a)]s_p(a). \end{aligned}\]Detach the coefficient $q(a)-\rho p(a)$ when using it in a log-probability surrogate. Do not renormalize the saved head to sum to one: that would erase tail mass. The actually sampled token can lie outside the head and still needs its own logged sampling probability for IS.
Exact zero drift now holds under $\widehat q$, not automatically under the actual $q$. Under actual sampling, the residual is $\mathbb E_q[s_p]-\mathbb E_{\widehat q}[s_p]$. Near-zero trainer tail mass also needs numerical handling; an epsilon prevents invalid division but does not make a wrong tail model correct. If the whole vocabulary is saved, use the full sum rather than computing an unnecessary $0/0$ tail ratio.
8. What the Evidence Supports—and What It Does Not
8. 实验支持什么，没有证明什么？
The paper compares corrections on shared group-centered REINFORCE training with SGD. It reports useful SC behavior under quantization and stronger SC-plus-IS behavior under severe staleness. Tests deliberately amplify mismatch and use short sequences; these are limits on extrapolation, not hidden guarantees for long agentic runs. See §5 and the limitations. The supplied discussion’s stability interpretation is consistent with those reported results.
Our independent calculations separate the claims:
| Question | Conclusion |
|---|---|
| Does exact SC remove the defined conditional drift? | Yes, when the mean score uses the actual sampler distribution. |
| Does it recover the current trainer’s gradient? | Not generally; the binary example already disproves it. |
| Is it guaranteed closer than the raw update? | No; drift and covariance error can cancel. |
| Does exact IS recover the target gradient? | Yes, with the right sampling object, support, and integrability. |
| Are clipped or token-local IS automatically exact? | No. Their scope and bias must be analyzed. |
| Does failure of target recovery imply no practical value? | No. Stability and finite-sample tradeoffs remain meaningful. |
The shortest intuition is this: SC recenters the arrows produced by the sampler; IS changes how frequently those arrows count toward the target average. Recentring can remove a harmful common push without relocating the entire average to the trainer’s objective. In the binary case, the distinction is visible in four numbers: target $0.09$, raw $0.08$, SC $0.16$, exact IS $0.09$.
That is why the debate should not be summarized as either “SC fixes off-policy gradients” or “SC does nothing.” A more defensible description is a drift-removing update with possible stability benefits, whose remaining target-gradient bias must be acknowledged.