Regularized Score Centering

A visual introduction to score centering and RSC.
从 score centering 到 RSC 的视频介绍。

Abstract

Asynchronous reinforcement learning increases training throughput, but data from stale policies and numerical mismatch between training and inference can destabilize learning. To stabilize off-policy learning, score centering (SC) removes the effect of reward baselines on the expected update, even without full importance sampling (IS). Our empirical study shows that full IS produces extreme gradients and that SC significantly reduces their magnitude. However, SC can still fail when the learner moves far from the behavior policy that generated the data. To limit this policy mismatch, we introduce Regularized Score Centering (RSC), which adds a reverse Kullback–Leibler (KL) penalty to encourage the learner’s next-token distribution to stay close to the behavior policy’s next-token distribution. RSC works with or without truncated importance sampling (TIS), which caps token importance weights. Our theoretical analysis supports this approach, showing that KL regularization improves convergence guarantees. We evaluate dense and mixture-of-experts models on mathematical reasoning and agentic tasks, including online shopping, tool use, games, and fully asynchronous software engineering. RSC with TIS substantially improves sample efficiency across all agentic benchmarks, attaining higher training rewards with the same number of sampled responses. In final evaluations, it achieves top performance across all six agentic tasks, improving by 14.3% on AppWorld and reaching 9.9× the strongest baseline’s score on BabyAI.

Note. This blog is an introductory version of the paper, focusing on an intuitive and empirical understanding of RSC. For theoretical proofs, please see the paper.

Note. 这篇 blog 是 paper 的 introductory version,重点通过直觉和实验理解 RSC。理论证明请阅读 paper。

Paper Figure 1. Adding the same reverse-KL penalty to SC and SC+TIS gives the two RSC variants. The three task panels show training rewards with three-seed means and sample standard deviations. Stronger methods retain their colors; the other curves are black.
论文 Figure 1。在 SC 与 SC+TIS 上加入同一类 reverse-KL penalty,得到两种 RSC 变体。三个任务面板展示训练 reward,曲线和阴影为三个 seed 的均值与样本标准差。表现较强的方法保留颜色,其余曲线显示为黑色。

Reinforcement learning and off-policy data

Imagine asking an assistant to buy a red mug within a budget. It searches, opens a product, checks its color, and decides whether to purchase it. The sequence of decisions is a trajectory, and a reward measures how well the outcome satisfies the request. Reinforcement learning uses these outcomes to change the assistant’s future decisions. A mathematical answer can receive a correctness reward, while an interactive task can provide feedback during the trajectory as well as at its end.

At each decision, the model has a history $h$: the request, its previous actions, and any observations it has received. Its policy assigns a probability to each possible next action $a$. For a language model, the elementary action is a token. A tool call or a sentence is built from several such decisions. We write the current policy as $\pi_\theta$, where $\theta$ denotes the model’s trainable parameters.

To improve the policy, training needs to know how changing those parameters changes an action’s probability. This sensitivity is called the score:

\[\psi_\theta(h,a)=\nabla_\theta\log \pi_\theta(a\mid h).\]

The symbol $\nabla_\theta$ means “take the gradient with respect to the model parameters.” A gradient tells an optimizer which parameter changes increase a quantity. Multiplying the score by a reward gives a basic policy-gradient update: reinforce choices associated with good outcomes. Here, score means a log-probability gradient, while an evaluation score measures task performance.

假设我们让一个助手购买预算以内的红色杯子。它需要搜索、打开商品、检查颜色,然后决定是否购买。连续做出的这些选择构成一条轨迹(trajectory),而奖励(reward)衡量最终结果是否满足要求。强化学习根据这些结果调整助手未来的行为。数学题可以用答案是否正确作为奖励,交互任务也可以在过程中提供反馈,并在结束时评价结果。

每次做决定时,模型看到一个历史(history) $h$:用户的请求、先前的动作,以及已经获得的观察。模型的策略(policy)为每个可能的下一步动作 $a$ 分配概率。对语言模型来说,最基本的动作是一个 token,一句话或一次工具调用由多个这样的决定组成。我们用 $\pi_\theta$ 表示当前策略,其中 $\theta$ 是模型可以训练的参数。

为了改善策略,训练过程需要知道:改变参数,会怎样改变某个动作的概率?这种敏感度称为 score:

\[\psi_\theta(h,a)=\nabla_\theta\log \pi_\theta(a\mid h).\]

$\nabla_\theta$ 表示对模型参数求梯度。梯度告诉优化器,沿哪些方向改变参数能够提高某个量。将 score 乘以奖励,就得到一种基本的 policy gradient 更新:加强与好结果相关的选择。这里的 score 是对数概率的梯度,而实验中的 evaluation score 才是任务表现的分数。

This update uses rewards from sampled actions, but generating that experience takes time. An agent may be waiting for a website, running a test suite, or finishing a long answer while training proceeds elsewhere. The model that started the trajectory can therefore differ from the model being updated when the reward arrives. We call the data-generating policy the behavior policy $\mu$ and the model being trained the learner $\pi_\theta$. Learning from data generated by a different policy is off-policy learning. Even freshly generated data can be off-policy if the generator uses older weights.

This distinction changes what a batch of experiences tells us. Suppose the behavior policy chooses between two actions equally often, but the learner now strongly prefers the better one. The batch still contains many examples of the other action. Giving every successful old action a positive push can pull the learner back toward the behavior policy, even when the learner already makes better choices.

One response is importance sampling (IS). It weights a sampled action by the ratio

\[\rho(h,a)=\frac{\pi_\theta(a\mid h)}{\mu(a\mid h)}.\]

A ratio above one means that the learner would choose this action more often than the behavior policy. Reweighting by this ratio corrects the current-action distribution at a fixed history. However, rare behavior actions can receive very large weights, allowing a few samples to dominate an update. The history and the rest of the trajectory still come from $\mu$. These are the two issues we will separate: how to use a sampled action’s reward, and how far the learner has moved from the experience that produced it.

From IS to SC and RSC: intuition and evidence

The methods below address three questions: how to use data from an older policy, how to keep a few tokens from dominating an update, and how to keep the learned policy close enough to the policy that supplied the data. We first examine one token at a fixed history $h$, since these corrections operate at the token level. The behavior policy $\mu$ stays fixed during an update; training combines the contributions across token positions.

下面围绕三个问题理解这些方法:怎样使用旧策略的数据,怎样避免少数 token 主导更新,以及怎样限制 learner 与提供数据的策略之间的差距。因为这些 correction 都在 token 层面进行,我们先固定一个 history $h$,只看下一个 token。更新时 behavior policy $\mu$ 保持不变,训练再将各位置的贡献汇总起来。

Step 1. Policy gradient: learn from rewarded choices

Policy gradient (PG) uses rewards to construct a gradient estimate. A sampled reward $R_i$ tells us how a response turned out, while its score $\psi_i=\psi_\theta(h,a_i)$ tells us how to change the selected token’s probability. Their product gives a basic contribution:

\[\widehat g_i=R_i\psi_i.\]

Here $i$ identifies a sample. With on-policy trajectories, averaging these contributions and summing across their tokens gives a policy-gradient estimate. An optimizer such as SGD then uses that estimate to update the parameters. PG specifies the gradient estimate; SGD specifies how to apply it.

When responses come from an older policy, the sampling distribution changes. We next examine how IS corrects the current token’s weight.

Policy Gradient(PG)利用 reward 构造梯度估计。一次采样的 reward $R_i$ 告诉我们回答的结果如何,score $\psi_i=\psi_\theta(h,a_i)$ 则告诉我们怎样改变所选 token 的概率。两者相乘,得到一个基本贡献:

\[\widehat g_i=R_i\psi_i.\]

这里 $i$ 标记一个样本。使用 on-policy 轨迹时,对这些贡献取平均并按 token 求和,就得到 policy-gradient 估计;随后,SGD 等优化器根据这个估计更新参数。PG 决定怎样估计梯度,SGD 决定怎样用它更新参数。

当回答来自较旧的策略时,采样分布就发生了变化。下面看 IS 怎样修正当前 token 的权重。

Step 2. IS corrects the current token’s weight

Importance sampling (IS) compensates for the difference between how often $\mu$ selects a token and how often $\pi_\theta$ would select it. It multiplies the contribution by $\rho_i=\pi_\theta(a_i\mid h)/\mu(a_i\mid h)$:

\[\widehat g_i^{\rm IS}=\rho_iR_i\psi_i.\]

In expectation, this changes the current action’s average from $\mu$ to $\pi_\theta$. It also explains why full IS needs no score centering to remove a reward baseline. A baseline has the same value for every action at a history, while the score’s mean under $\pi_\theta$ is zero:

\[\mathbb E_{\mu}[\rho\psi_\theta]=\mathbb E_{\pi_\theta}[\psi_\theta]=\nabla_\theta\sum_a \pi_\theta(a\mid h)=0.\]

Multiplying this zero by the baseline still gives zero, so subtracting any such baseline leaves the expected IS update unchanged. This identity corrects the current action’s weighting; the histories and subsequent decisions in the data still follow $\mu$.

Importance sampling(IS)用一个权重补偿采样频率的差别:如果某个 token 在 $\mu$ 下比在 $\pi_\theta$ 下更少出现,就提高它的权重;反之则降低。具体来说,用 $\rho_i=\pi_\theta(a_i\mid h)/\mu(a_i\mid h)$ 乘上原来的贡献:

\[\widehat g_i^{\rm IS}=\rho_iR_i\psi_i.\]

在期望中,这会把当前动作的平均从按 $\mu$ 计算变为按 $\pi_\theta$ 计算。这也解释了为什么完整 IS 不需要 score centering 就能去掉 reward baseline 的贡献:在同一个 history 下,baseline 对所有动作相同,而 score 在 $\pi_\theta$ 下的均值为零:

\[\mathbb E_{\mu}[\rho\psi_\theta]=\mathbb E_{\pi_\theta}[\psi_\theta]=\nabla_\theta\sum_a \pi_\theta(a\mid h)=0.\]

baseline 乘上这个零,贡献仍然为零。因此,从 reward 中减去任何这样的 baseline,都不改变 IS 的期望更新。但这个恒等式只修正当前动作的权重,数据中的 histories 和后续决定仍然来自 $\mu$。

Optional derivation: why the IS weighting works

At a fixed history $h$, $Q^{\mu}(h,a)$ is the expected reward after choosing action $a$ and then following $\mu$. To see how IS changes the current action’s weighting, expand the expectation as a probability-weighted sum. The sampling probability $\mu(a\mid h)$ cancels the denominator of $\rho$:

\[\begin{aligned} \mathbb E_{\mu}[\widehat g^{\rm IS}] &=\sum_a \mu(a\mid h)\frac{\pi_\theta(a\mid h)}{\mu(a\mid h)}Q^{\mu}(h,a)\psi_\theta(h,a)\\ &=\sum_a \pi_\theta(a\mid h)Q^{\mu}(h,a)\psi_\theta(h,a)\\ &=\mathbb E_{a\sim \pi_\theta}[Q^{\mu}(h,a)\psi_\theta(h,a)]. \end{aligned}\]

This requires $\mu(a\mid h)>0$ wherever $\pi_\theta(a\mid h)>0$. The final line averages the current action under $\pi_\theta$, while $Q^{\mu}$ still evaluates what happens afterward under $\mu$. Histories in the training data also remain those generated by $\mu$.

The next six tasks involve actions and environment feedback. WebShop asks the model to shop, TextWorld Cooking to prepare food, AppWorld to operate applications, BabyAI to navigate and manipulate objects, and Crafter and Craftax to gather resources and survive. Rewards measure success or progress within each task. We use agentic environments because a change in action can redirect later observations and rewards, making behavior trajectories less representative of what the learner would do.

IS corrects the current token’s weight, yet the curves remain unstable: WebShop loses part of its early gains, AppWorld’s reward declines, and BabyAI stays weak. A few tokens with very large ratios can dominate an update. This motivates limiting the multiplier.

接下来的六个任务都有动作和 environment feedback。 WebShop 让模型购物,TextWorld Cooking 要求完成烹饪任务,AppWorld 要求操作应用,BabyAI 涉及导航和物体操作,Crafter 与 Craftax 则需要收集资源并生存。各任务的 reward 衡量完成情况或进展。我们使用 agentic 环境,是因为动作的改变会影响后续观察和 reward,让 behavior 产生的轨迹更难代表 learner 自己执行时的结果。

IS 修正了当前 token 的权重,但这些曲线仍不稳定:WebShop 丢失了部分早期提升,AppWorld 的 reward 下降,BabyAI 则持续较弱。少数概率比很大的 token 可能主导整个更新,因此我们希望限制这个乘数。

Raw IS training rewards across six agentic tasks. Each panel uses the task’s own reward scale.
六个 agentic 任务上的原始 IS 训练 reward。每幅图使用对应任务的 reward 尺度。

Step 3. TIS limits the multiplier, but changes the mean update

Truncated importance sampling (TIS) caps the ratio at $c$ to reduce the effect of large weights:

\[\widehat g_i^{\rm TIS}=\min(\rho_i,c)R_i\psi_i.\]

This is the weighting used by CISPO. The cap reduces ratio-driven variance, but introduces bias: beyond the cap, the weights no longer turn an average under $\mu$ into an average under $\pi_\theta$. Consequently, the score’s weighted mean need not be zero, and a reward baseline can affect the expected update again.

A cap alone does not sustain learning in these experiments. TIS falls toward zero on WebShop and AppWorld, stays weak on BabyAI, and loses earlier gains on the other tasks. The curves show why we need more than a bounded importance weight; they do not isolate truncation bias as the sole cause of these failures.

Truncated importance sampling(TIS)把概率比限制在上界 $c$,减小大权重的影响:

\[\widehat g_i^{\rm TIS}=\min(\rho_i,c)R_i\psi_i.\]

这就是 CISPO 使用的加权方式。截断减少了大概率比带来的 variance,却引入了 bias:超过上界后,这些权重不能再把按 $\mu$ 的平均变成按 $\pi_\theta$ 的平均。因此,加权 score 的均值不再保证为零,reward baseline 又可能影响期望更新。

这些实验中,仅设上界还不足以维持学习。 TIS 在 WebShop 和 AppWorld 上降到接近零,在 BabyAI 上持续较弱,其他任务也丢失了早期提升。这些曲线说明,限制 importance weight 还不够;它们本身并没有单独验证 truncation bias 是失败的唯一原因。

IS and TIS across six agentic tasks. Lines and bands show mean training reward and sample standard deviation across three seeds. Reward scales differ by task.
六个 agentic 任务上的 IS 与 TIS 对照。曲线与阴影为三个 seed 的训练 reward 均值和样本标准差;各任务的 reward 尺度不同。

We next remove the part of the update caused by giving every action the same baseline credit.

下一步要去掉的,是给所有动作相同的 baseline 评价所带来的更新。

Step 4. SC removes the common reward contribution

At the same history, we can compare an action’s expected reward with the policy’s average expected reward over its next actions. This difference is the action’s advantage, denoted by $A$. We want the update to reflect each action’s advantage.

This comparison depends on how the policy continues after the action. When the subsequent rollout follows the behavior policy $\mu$, and the baseline also averages next actions under $\mu$, we write the advantage as $A^{\mu}$. The superscript tells us which policy supplies those future decisions and the baseline.

Score centering subtracts the mean weighted score before multiplying by the reward. Writing $\rho^{\rm TIS}=\min(\rho,c)$, SC+TIS uses

\[\widehat g_i^{\rm SC+TIS}=R_i\bigl(\rho_i^{\rm TIS}\psi_i-\mathbb E_{\mu}[\rho^{\rm TIS}\psi]\bigr).\]

The mean is taken over possible next tokens at the same history.

Let $Q^{\mu}(h,a)$ be the expected reward after choosing $a$ and continuing with $\mu$, let $V^{\mu}(h)=\mathbb E_{a\sim \mu}[Q^{\mu}(h,a)]$, and let $A^{\mu}=Q^{\mu}-V^{\mu}$. At a fixed history, the mean sampled reward equals $V^{\mu}$. With the exact center,

\[\begin{aligned} \mathbb E_{\mu}[\widehat g^{\rm SC+TIS}] &=\mathbb E_{\mu}[Q^{\mu}\rho^{\rm TIS}\psi]-V^{\mu}\mathbb E_{\mu}[\rho^{\rm TIS}\psi]\\ &=\mathbb E_{\mu}[(Q^{\mu}-V^{\mu})\rho^{\rm TIS}\psi]\\ &=\mathbb E_{\mu}[A^{\mu}\rho^{\rm TIS}\psi]. \end{aligned}\]

Setting every weight to one gives $\mathbb E_{\mu}[\widehat g^{\rm SC}]=\mathbb E_{\mu}[A^{\mu}\psi]$. Centering cancels the common-value term while preserving the chosen reward weights and behavior continuations.

SC therefore changes TIS’s expected update from $Q^{\mu}$-weighted to $A^{\mu}$-weighted, while $A^{\mu}$ still depends on behavior continuations. This identity describes the expected update; implementing it requires no critic or explicit estimate of $Q^{\mu}$, $V^{\mu}$, or $A^{\mu}$.

So yes, we can omit IS. The difference is which expectation we estimate. Both methods use data sampled from $\mu$. After removing the baseline contribution, SC without IS still estimates the advantage-weighted score mean under $\mu$. Full IS uses the ratio to estimate the current action’s average contribution under $\pi_\theta$ from those same behavior samples. SC+TIS retains capped weights, so it only partially corrects this weighting; the histories and continuations still come from $\mu$.

How does SC compare with the methods introduced earlier? Across the six agentic tasks below, SC+TIS sustains much higher rewards than TIS on several tasks, and both centered variants learn well on TextWorld. BabyAI remains weak, pointing to a limitation beyond the common-value contribution.

与前面介绍的方法相比,SC 表现怎样? 下面回到六个 agentic 任务:SC+TIS 在多个任务上持续取得明显高于 TIS 的 reward,两种 centered 方法在 TextWorld 上都能较好地学习。但 BabyAI 仍然较弱,提示公共价值贡献之外还有问题。

IS, TIS, SC, and SC+TIS across six tasks. Three-seed means and sample standard deviations. TIS uses cap 5 and SC+TIS uses cap 2.
六个任务上的 IS、TIS、SC 与 SC+TIS 对照。曲线为三个 seed 的均值与样本标准差。TIS 的 cap 为 5,SC+TIS 的 cap 为 2。

How we estimate the center also matters. Monte Carlo centering independently samples auxiliary next tokens from $\mu$ at the same history and averages their weighted scores. These tokens need no new rollouts or reward evaluations. This estimate is unbiased, whereas Top-128 approximations retain bias by using only the most likely tokens, with a modeled tail in the Countdown version.

The following comparisons use Countdown and mathematical reasoning. In Countdown, the model must form an expression that reaches a target using each of four supplied numbers exactly once; a valid answer receives reward 1 and an invalid answer 0. The mathematical reasoning tasks similarly use a verifier to check the generated answer. Every tested Monte Carlo variant finishes with a higher held-out evaluation than its corresponding Top-128 baseline. Thus accurate centering improves learning. The remaining failures motivate controlling policy mismatch.

中心怎样估计,也会影响学习。 Monte Carlo centering 在同一个 history 下,从 $\mu$ 独立抽取辅助下一 token,再对它们的加权 score 取平均。这些 token 不需要新的 rollout 或 reward 评估。这个估计是无偏的;Top-128 则使用概率最高的 token 近似中心,仍有 bias,Countdown 的版本还对剩余概率质量进行建模。

下面在 Countdown 与数学推理任务上比较这些估计方法。Countdown 给出四个数字和一个目标值,模型需要写出一个达到目标的算式,每个数字恰好使用一次;有效答案得到 reward 1,否则为 0。数学推理任务同样由验证程序检查生成的答案。每个测试的 Monte Carlo 变体,其最终 held-out evaluation 都高于对应的 Top-128 baseline。因此,准确估计中心有助于学习。仍然出现的失败促使我们进一步控制 policy mismatch。

Center-estimator comparison: training rewards and separate learner evaluations. Math uses one seed per estimator and rollout precision; Countdown uses three. Its Top-128 baseline includes a modeled tail.
中心估计方法对照:分别展示训练 reward 与 learner 评估。数学任务每种估计器、每种 rollout 精度使用一个 seed;Countdown 使用三个 seed,其 Top-128 baseline 包含建模的 tail。

Step 5. RSC controls the mismatch that centering leaves

SC removes the common reward contribution, leaving an advantage-weighted update. But $A^{\mu}$ still evaluates a token using future decisions under $\mu$. Making the same comparison with rollout and baseline under the learner gives $A^{\pi_\theta}$, which can differ for the same token at the same history. This matters when a tool call changes the next observation or a token starts a solution that the two policies complete differently.

RSC adds reverse KL to encourage the learner’s next-token distribution to stay close to the behavior policy’s distribution. This limits policy movement during fitting; refreshing the behavior policy then updates the data and continuation values used by the next round.

To connect that idea to the curves, define the surrogate objective $\mathcal W_{\mu}$ as the expected sum of token contributions $w_t$:

\[\begin{aligned} \mathcal W_{\mu}(\pi_\theta)&=\mathbb E_{\mu}\!\left[\sum_t w_t(\rho_t)\right],\\ w_t(\rho_t)&=A_t^{\mu}F_a(\rho_t)-\beta(\rho_t\log\rho_t-\rho_t+1). \end{aligned}\]

The first term is the reward surrogate: $F_a$ corresponds to the chosen reward weight, and becomes $\log\rho$ for SC without IS. The second term is the KL penalty, with strength $\beta\ge0$. At each history, the parenthesized expression averages to reverse KL. Thus $w_t$ is one token’s contribution, called the surrogate integrand; summing it and taking the expectation gives the full regularized objective. We hold $\mu$ and its advantages fixed when differentiating. Setting $\beta=0$ gives SC variants, while $\beta>0$ gives RSC variants.

SC 去掉公共 reward 的贡献,留下 advantage 加权的更新。但 $A^{\mu}$ 仍然按 $\mu$ 的后续决定评价当前 token。 如果 rollout 和 baseline 都改用 learner $\pi_\theta$,就得到 $A^{\pi_\theta}$;即使是同一 history 下的同一个 token,两者也可能不同。一次工具调用改变后续观察,或者某个 token 开始了一条两种策略会以不同方式完成的解题路径时,这种差别尤其重要。

RSC 加入 reverse KL,鼓励 learner 的下一 token 分布保持接近 behavior policy。这样可以限制 fitting 期间的策略移动;刷新 behavior policy 后,下一轮使用的数据和 continuation values 也随之更新。

为了把这个想法和曲线联系起来,我们把 surrogate objective $\mathcal W_{\mu}$ 定义为各 token 贡献 $w_t$ 之和的期望:

\[\begin{aligned} \mathcal W_{\mu}(\pi_\theta)&=\mathbb E_{\mu}\!\left[\sum_t w_t(\rho_t)\right],\\ w_t(\rho_t)&=A_t^{\mu}F_a(\rho_t)-\beta(\rho_t\log\rho_t-\rho_t+1). \end{aligned}\]

第一项是 reward surrogate:$F_a$ 对应所选的 reward weight,不带 IS 的 SC 使用 $F_a(\rho)=\log\rho$。第二项是 KL penalty,强度由 $\beta\ge0$ 控制,括号内表达式在固定 history 下的期望就是 reverse KL。所以,$w_t$ 是一个 token 对目标的贡献,称为 surrogate integrand;将它按 token 求和并取期望,才得到完整的正则化目标。求导时固定 $\mu$ 及其 advantage。$\beta=0$ 对应 SC variants,$\beta>0$ 对应 RSC variants。

Optional derivation: reward weights and the surrogate

Write $a(\rho)$ for the reward weight: $\rho$ for IS, $1$ for SC without IS, and $\min(\rho,c)$ for SC+TIS. Here $a$ denotes an action and $a(\cdot)$ denotes the reward-weight function. Integrating the weight divided by the ratio gives

\[F_a(\rho)=\int_1^\rho\frac{a(u)}{u}\,du,\qquad \nabla_\theta[AF_a(\rho)]=Aa(\rho)\psi_\theta.\]

For IS, $F_a(\rho)=\rho-1$, so the reward-surrogate gradient is $A\rho\psi_\theta$. For SC without IS, $F_a(\rho)=\log\rho$, giving $A\psi_\theta$. For SC+TIS, it gives $A\min(\rho,c)\psi_\theta$. These match the corresponding expected updates, with advantage and behavior distribution fixed.

The KL part follows from $\mathbb E_{\mu}[\rho]=1$:

\[\mathrm{KL}(\pi_\theta\|\mu)=\sum_a \pi_\theta(a\mid h)\log\rho=\mathbb E_{\mu}[\rho\log\rho-\rho+1].\]

Step 6. Read stability from the gradient curves

The model’s logit $z_a$ is the raw value that softmax converts into a token probability. Differentiating $w$ with respect to $z_a$ tells us how strongly the objective pushes that value. Below, $A=A^{\mu}$ is the fixed behavior advantage. The plot and table show the same derivative: the table includes the common factor $1-\pi_\theta(a\mid h)$, while the plot removes it to make the methods’ dependence on $\rho$ easier to compare.

模型的 logit $z_a$ 是 softmax 转换为 token 概率之前的原始数值。对 $w$ 关于 $z_a$ 求导,可以看出目标有多强地推动这个数值改变。下文用 $A=A^{\mu}$ 表示固定的 behavior advantage。图和表展示同一个导数:表格保留公共因子 $1-\pi_\theta(a\mid h)$,图中则提取这个因子,让各方法对 $\rho$ 的依赖更容易比较。

Method \(\partial w/\partial z_a\)
IS \([1-\pi_\theta(a\mid h)]A\rho\)
SC+TIS \([1-\pi_\theta(a\mid h)]A\min(\rho,c)\)
RSC (TIS) \([1-\pi_\theta(a\mid h)][A\min(\rho,c)-\beta\rho\log\rho]\)
SC (w/o IS) \([1-\pi_\theta(a\mid h)]A\)
RSC (w/o IS) \([1-\pi_\theta(a\mid h)][A-\beta\rho\log\rho]\)
Paper Figure 2, rendered interactively. The plot shows \((\partial w/\partial z_a)/(1-\pi_\theta(a\mid h))\); the table includes the common factor. Defaults: \(A=1\), \(c=2\), and \(\beta=0.05\), with standard, uncapped reverse KL. Adjust \(c\) and \(\beta\) to compare the curves.
论文 Figure 2 的交互版本。图中画的是 \((\partial w/\partial z_a)/(1-\pi_\theta(a\mid h))\),表格则保留公共因子。默认 \(A=1\)、\(c=2\)、\(\beta=0.05\),使用标准、未截断的 reverse KL。可以调整 \(c\) 和 \(\beta\) 比较曲线。

IS keeps the multiplier $\rho$; SC removes it; SC+TIS caps it. That is the main contrast in the plot. RSC also subtracts $\beta\rho\log\rho$: to the right of $\rho=1$, it opposes further probability increases; to the left, it pushes in the other direction. The plotted coefficients describe the corresponding expected updates; the empirical plots below show individual sampled gradients.

Does the same pattern appear in actual model gradients? A large ratio is possible only when $\mu(a\mid h)$ is small, since $\pi_\theta(a\mid h)\le1$. We therefore plot measured norms against the behavior probability $\mu$.

IS 保留乘数 $\rho$,SC 去掉它,SC+TIS 则限制它的上界。 这是曲线最主要的区别。RSC 还减去 $\beta\rho\log\rho$:在 $\rho=1$ 右侧,它抑制概率继续增大;在左侧,它则向相反方向推动。这些系数描述相应的期望更新;下面的实验图展示各个采样梯度。

实际模型的梯度也会呈现这种差别吗?因为 $\pi_\theta(a\mid h)\le1$,只有 $\mu(a\mid h)$ 较小时,才可能出现很大的概率比。因此,下面以 behavior 概率 $\mu$ 为横轴,直接展示测得的梯度范数。

Measure both logit and parameter gradients

Figure 3 compares the five update rules on the same checkpoint and token batch, holding sampled actions, response advantages, and auxiliary Monte Carlo draws fixed. The first row measures each token’s output-logit gradient norm; the second measures its gradient norm over all model parameters. SC and SC+TIS have smaller extreme gradients than IS in both spaces.

Figure 3 在同一个 checkpoint 和同一批 token 上比较五种更新规则,固定采样动作、response advantage 和辅助 Monte Carlo 样本。第一行是每个 token 的输出 logit 梯度范数,第二行是它在全模型参数上的梯度范数。两个空间中,SC 与 SC+TIS 的极端梯度都小于 IS。

Paper Figure 3. Both rows use behavior probability on the horizontal axis. Colored points show the measured token gradient norms; black lines mark the 99th percentile within each probability bin. Logit norms use the paper’s 0–5 range and parameter norms use 0–10⁴. Select methods or a row to inspect the distributions. RSC uses \(\beta=0.1\) in this diagnostic.
论文 Figure 3。两行均以 behavior 概率为横轴,彩色散点展示实际测得的 token 梯度范数,黑线标出各概率区间内的第 99 百分位数。Logit 范数沿用论文的 0–5 范围,参数范数为 0–10⁴。可选择方法或某一行查看分布。该诊断中 RSC 使用 \(\beta=0.1\)。

Adding KL also lowers the low-probability logit-gradient P99 for both centered variants in this batch. These measurements connect the analytical curves to actual gradients. The next section examines what happens across repeated updates, as staleness and agentic interaction make the remaining policy mismatch matter.

在这批数据中,加入 KL 还降低了两种 centered 方法在低 behavior 概率 token 上的 logit 梯度 P99。这些测量把解析曲线和实际梯度联系起来。下一节再看多次更新累积后的结果,以及 staleness 和 agentic interaction 怎样放大剩余 policy mismatch 的影响。

Failure Modes of SC and the Role of RSC in Practice

We now bring together the results on the reasoning and interactive tasks introduced above. Learning curves show how quickly each method improves, while held-out evaluations test the resulting learner on held-out task instances.

We compare RSC with SC, SC+TIS and several importance-sampling baselines, including IS-PPOClip, which combines importance weighting with a PPO-clipped update. RSC with TIS achieves leading final performance across all six agentic tasks. Its gains are especially large on BabyAI and Crafter: compared with IS-PPOClip, the strongest baseline outside the RSC family on these tasks, its score is $9.9\times$ as high on BabyAI and 52% higher on Crafter. It also improves on the strongest established baselines by 16% on Craftax and 14.3% on AppWorld.

现在将前面介绍的推理和交互任务上的结果放在一起。训练曲线反映各方法改善得有多快,held-out evaluation 则在 held-out task instances 上检验训练后的 learner。

我们将 RSC 与 SC、SC+TIS 以及多种 importance sampling baseline 比较,其中 IS-PPOClip 将 importance weighting 与 PPO clipping 相结合。带 TIS 的 RSC 在六个 agentic 任务上都取得领先的最终表现。 BabyAI 和 Crafter 上的提升尤其显著:相比这两个任务中最强的非 RSC baseline IS-PPOClip,其分数在 BabyAI 上达到 $9.9\times$,在 Crafter 上高出 52%。在 Craftax 和 AppWorld 上,它也分别比最强的已有 baseline 提高 16% 和 14.3%。

Why test RSC in agentic environments?

Agentic interaction can amplify off-policyness through its effect on future decisions and rewards. At the same history, $A^{\mu}$ evaluates an action through behavior continuations, while $A^{\pi_\theta}$ evaluates it through learner continuations. In an interactive task, a difference in action can change the next observation, the choices available afterward, and the eventual outcome. Those changes can widen the gap between $A^{\mu}$ and $A^{\pi_\theta}$, even at a similar sampler staleness. SC removes the common-value contribution but still uses $A^{\mu}$, so its update can favor a choice whose advantage is lower under learner continuations. This is why we use agentic tasks to test whether RSC helps when behavior-based estimates become less representative of the learner.

Figure 5 compares mathematical reasoning with Crafter at the same sampler refresh interval, $R=64$. In math, the model generates a solution and receives an answer-correctness reward. Crafter adds environment feedback between actions, letting us compare the benefit of KL across these task structures.

Agentic interaction 会通过后续决定和 reward 放大 off-policyness 的影响。 在同一个 history 下,$A^{\mu}$ 根据 behavior 的后续 rollout 评价动作,$A^{\pi_\theta}$ 则根据 learner 的后续 rollout 评价动作。交互任务中,动作的差别会改变下一次观察、之后可选的动作,以及最终结果。因此,即使 sampler staleness 相近,$A^{\mu}$ 与 $A^{\pi_\theta}$ 的差距也可能更大。SC 去掉了公共价值的贡献,却仍然使用 $A^{\mu}$,所以它推动的选择在 learner 自己的后续 rollout 下可能没有那么好。我们用 agentic 任务做实验,正是为了检验:当 behavior 上的估计更难代表 learner 的结果时,RSC 是否能改善学习。

Figure 5 在相同的 sampler 刷新间隔 $R=64$ 下比较数学推理与 Crafter。数学任务让模型生成解答,再根据答案正误给出 reward;Crafter 则在动作之间提供 environment feedback,让我们比较这两种任务结构下 KL 的作用。

Paper Figure 5. Each setting occupies one row: training performance followed by the signed mean log-probability gap, \(\log \pi_\theta-\log \mu\). Both use \(R=64\). Curves and bands show three-seed means and sample standard deviations.
论文 Figure 5。每个 setting 占一行,依次展示训练表现和带符号的平均 log-probability gap:\(\log \pi_\theta-\log \mu\)。两者均使用 \(R=64\),曲线和阴影为三个 seed 的均值与样本标准差。

SC and SC+TIS sustain learning on math, while RSC with TIS learns substantially better than SC+TIS on Crafter. In an interactive environment, gathering a resource or choosing a route can open different future choices. An action’s value therefore depends on how subsequent decisions use those opportunities. This helps explain why behavior advantages can become less representative of learner outcomes. The tasks and models differ, so the next experiment varies staleness within each setting to examine policy separation more directly.

SC 与 SC+TIS 在数学任务上能够持续学习,而在 Crafter 上,RSC with TIS 的学习表现明显优于 SC+TIS。 在交互环境中,收集一种资源或选择一条路线,会开启不同的后续选择。因此,一个动作的价值取决于后续决策如何利用这些机会。这有助于理解,为什么 behavior advantage 可能更难代表 learner 的实际结果。这里的任务和模型不同;接下来,我们在各自 setting 内改变 staleness,更直接地考察策略分离的影响。

When the data fall further behind

Task structure changes the consequences of different continuations; sampler staleness allows the policies to move further apart. We control staleness by refreshing the behavior policy only once every $R$ learner updates. Increasing $R$ lets the learner move further before the model collecting the next answers catches up. The Countdown experiment keeps the model, three-seed cohort, and training budget fixed while increasing this refresh interval.

任务结构改变了不同 continuation 所带来的后果,sampler staleness 则让两个策略有机会进一步分离。我们通过每隔 $R$ 次 learner 更新才同步一次 behavior policy,来控制 staleness。$R$ 越大,learner 就能在收集下一批答案的模型追上来之前走得更远。Countdown 实验固定模型、三个 seed 和训练预算,逐步增大这个刷新间隔。

Countdown uses refresh intervals $R=16,32,65$; MoE math uses $R=64,256,512$. Each pair shows reward and policy gap. Countdown uses three seeds; MoE uses two, two, and three seeds respectively. The signed Countdown gap and absolute MoE gap are different metrics.
Countdown 的刷新间隔为 $R=16,32,65$,MoE 数学任务为 $R=64,256,512$。每组展示 reward 与 policy gap。Countdown 使用三个 seed,MoE 各间隔分别使用两个、两个、三个 seed。Countdown 展示带符号差异,MoE 展示绝对差异。

The paper’s separate learner evaluation shows the same contrast. SC works well when refreshes are frequent, then loses most of its performance at the largest interval. RSC (TIS) retains strong held-out performance. SC+TIS also remains strong in this experiment, so the full comparison shows both the benefit of bounded importance weighting and the behavior of regularization. Unit-weight RSC improves greatly over SC at the largest interval, but retains more seed variation than RSC (TIS).

论文中单独对 learner 做的评估,也显示出这个差别。刷新频繁时,SC 能学得很好;间隔最大时,它的表现大幅下降,而 RSC (TIS) 仍保持较好的 held-out performance。 这个实验中的 SC+TIS 也保持良好,因此完整对照同时呈现了有界 importance weighting 的作用与 regularization 的表现。间隔最大时,单位权重 RSC 也明显优于 SC,但 seed 之间的差异比 RSC (TIS) 更大。

Learning curves across all six tasks

The training curves show how reward changes as each method collects more experience. Within each task, the methods use the same number of sampled responses per update, so higher reward at the same update count means better sample efficiency. RSC with TIS learns significantly more efficiently, with particularly clear gains on Crafter and Craftax: it stays substantially ahead of SC, SC+TIS, and IS-PPOClip through much of training. The final evaluations then check whether this advantage carries over to the trained learner.

训练曲线展示各方法随着采样经验增加,reward 如何变化。同一任务中,各方法每次更新使用相同数量的采样回答,因此,相同更新次数下取得更高 reward,就意味着更高的采样效率。带 TIS 的 RSC 显著提升了采样效率,Crafter 与 Craftax 上的优势尤其清楚:在训练的大部分阶段,它明显领先于 SC、SC+TIS 和 IS-PPOClip。后面的最终评估则检查这一优势是否延续到了训练后的 learner。

Training rewards under the behavior policy. Curves show three-seed means with bands for sample standard deviation. Crafter and Craftax use cumulative environmental rewards; the other tasks use their respective reward scales.
行为策略上的训练 reward。曲线展示三个 seed 的均值,阴影为样本标准差。Crafter 与 Craftax 使用累积环境奖励,其余任务采用各自的 reward 尺度。

Final evaluation: every method and task

The table reports held-out checkpoint evaluation scores as the mean ± sample standard deviation across three seeds. WebShop measures shopping reward, TextWorld the fraction of possible cooking points, AppWorld task completion, and BabyAI step-discounted terminal reward. These scores use a percentage scale. Crafter reports its geometric achievement score, while Craftax reports total episode reward. Compare methods within each task. Bold follows the paper: the highest mean, plus the second highest if it is within 5% of the highest.

表格报告 held-out tasks 上的 checkpoint 评估分数,展示三个 seed 的均值与样本标准差。WebShop 衡量购物 reward,TextWorld 衡量烹饪得分占满分的比例,AppWorld 衡量任务完成率,BabyAI 使用按完成步数折扣的终止奖励,这四项以百分数表示。Crafter 报告几何平均 achievement 分数,Craftax 报告整个 episode 的累积 reward。各方法应在同一任务内部比较。粗体与论文一致:最高均值,以及与最高值相差不超过 5% 的第二名。

Method WebShop TextWorld AppWorld BabyAI Crafter Craftax
IS 40.94± 36.13 21.77± 16.42 6.43± 11.14 0.13± 0.13 1.02± 0.70 0.83± 0.96
Seq-MIS 47.25± 40.95 96.95± 3.22 7.02± 5.26 1.71± 1.75 1.93± 0.82 2.39± 1.43
TIS 0.00± 0.00 19.77± 13.45 0.00± 0.00 0.00± 0.00 0.57± 0.44 0.39± 0.63
IS-PPOClip 74.81± 0.28 99.73± 0.25 9.36± 3.65 1.83± 1.36 9.34± 0.46 4.04± 0.27
SC 41.42± 35.57 99.06± 0.80 12.28± 1.75 0.80± 0.95 1.12± 1.12 1.58± 1.10
SC+TIS 52.21± 30.06 92.14± 13.62 12.28± 4.64 0.30± 0.32 3.27± 1.89 3.20± 0.58
RSC (w/o IS) 60.46± 9.02 95.29± 4.32 11.11± 3.65 7.32± 5.80 0.94± 0.29 1.46± 0.64
RSC (TIS) 74.80± 1.15 99.95± 0.08 14.04± 1.75 18.06± 1.77 14.21± 3.40 4.71± 0.03

The comparison with SC+TIS is also strong: adding KL raises its WebShop evaluation score by 43%, and RSC with TIS scores close to the maximum on TextWorld. Together, the learning curves and held-out evaluations show improvements in both sample efficiency and final performance.

与 SC+TIS 的比较同样明显:加入 KL 后,WebShop 的评估分数提高 43%,带 TIS 的 RSC 在 TextWorld 上也接近满分。结合训练曲线与 held-out evaluation,可以同时看到采样效率和最终表现的提升。

Fully asynchronous software engineering

In the software-engineering experiments, the model receives a task description and a code repository. It can inspect files, edit code, and run tests; a test harness checks whether its final changes solve the task and provides the success reward. We train on SWE-Gym tasks and evaluate on SWE-bench Verified. Completed trajectories enter the training queue while other agents continue editing files or running tests. The learner keeps changing during this work, so policy lag emerges naturally. We compare two configurations: A uses a learning rate of $10^{-6}$, and B uses $2\times10^{-6}$.

软件工程实验给模型一段任务描述和一个代码仓库。模型可以查看文件、修改代码、运行测试,最后由测试程序判断改动是否解决任务,并给出成功与否的 reward。训练使用 SWE-Gym 任务,评估使用 SWE-bench Verified。在训练过程中,完成的轨迹进入训练队列,其他 agent 同时继续编辑文件或运行测试。learner 在这个过程中不断更新,因此策略延迟会自然产生。我们比较两种配置:A 的学习率为 $10^{-6}$,B 的学习率为 $2\times10^{-6}$。

Paper Figure 8. Each learning-rate setting occupies one row, with training reward followed by the mean absolute log-probability gap $|\log\pi_\theta-\log\mu|$. Each curve is one run, smoothed with EMA $\alpha=0.05$.
论文 Figure 8。每个 learning-rate setting 占一行,依次展示训练 reward 和对数概率绝对差异的均值 $|\log\pi_\theta-\log\mu|$。每条曲线对应一个 run,使用 $\alpha=0.05$ 的 EMA 平滑。

An RSC variant achieves the highest evaluation score in each configuration. The advantage from adding KL is much larger in B for both versions of SC: RSC without IS improves on SC by 127%, and RSC with TIS improves on SC+TIS by 51%. This stronger benefit coincides with a larger learner–behavior probability gap in the curves. It supports the central motivation: regularization matters more when behavior data become less representative of the learner.

The table reports SWE-bench Verified Average@8 (%): each task is attempted eight times, and the binary success outcomes are averaged across attempts and tasks.

两种配置中都有一个 RSC 变体取得最高评估分数。 对 SC 的两种版本而言,加入 KL 在 B 中的收益都比 A 大得多:不带 IS 的 RSC 比 SC 提高 127%,带 TIS 的 RSC 比 SC+TIS 提高 51%。训练曲线中,B 也具有更大的 learner–behavior 概率差异。这与核心动机一致:当 behavior 数据更难代表 learner 的行为时,正则化更有价值。

表格报告 SWE-bench Verified 的 Average@8(%):每个任务执行八次,再对这些执行的二元成功结果和所有任务取平均。

Method A: LR $10^{-6}$ B: LR $2\times10^{-6}$
IS 57.08 0.40
TIS 52.33 51.51
SC 58.48 21.13
SC+TIS 59.06 36.04
RSC (w/o IS) 61.14 47.89
RSC (TIS) 54.77 54.47

The idea to take away

Learning from an older policy requires care in both credit assignment and policy movement. SC removes a common-value contribution from the expected update. TIS limits ratio amplification in the reward weights. KL keeps the learner’s action distribution close to the behavior policy that supplies its experience. RSC puts these pieces together in one family of updates. The resulting gains are particularly clear in interactive tasks, where many decisions separate an initial action from its outcome.

For an earlier, focused introduction to the centering mechanism, see Score Centering under Policy Mismatch. The present article accompanies our work Regularized Score Centering.

利用旧策略产生的经验学习,需要同时处理动作贡献和策略变化。SC 从期望更新中去掉公共价值的贡献,TIS 限制 reward 权重中的比值放大,KL 则让 learner 的动作分布与提供经验的 behavior policy 保持接近。 RSC 将这些部分组合成一个统一的方法族。在动作与最终结果之间隔着许多决策的交互任务中,这种组合带来了尤其明显的收益。

关于 centering 机制更集中的介绍,可以阅读此前的 Score Centering under Policy Mismatch。本文配合我们的工作 Regularized Score Centering。