Regularized Score Centering
Abstract
Asynchronous reinforcement learning increases training throughput, but data from stale policies and numerical mismatch between training and inference can destabilize learning. To stabilize off-policy learning, score centering (SC) removes the effect of reward baselines on the expected update, even without full importance sampling (IS). Our empirical study shows that full IS produces extreme gradients and that SC significantly reduces their magnitude. However, SC can still fail when the learner moves far from the behavior policy that generated the data. To limit this policy mismatch, we introduce Regularized Score Centering (RSC), which adds a reverse Kullback–Leibler (KL) penalty to encourage the learner’s next-token distribution to stay close to the behavior policy’s next-token distribution. RSC works with or without truncated importance sampling (TIS), which caps token importance weights. Our theoretical analysis supports this approach, showing that KL regularization improves convergence guarantees. We evaluate dense and mixture-of-experts models on mathematical reasoning and agentic tasks, including online shopping, tool use, games, and fully asynchronous software engineering. RSC with TIS substantially improves sample efficiency across all agentic benchmarks, attaining higher training rewards with the same number of sampled responses. In final evaluations, it achieves top performance across all six agentic tasks, improving by 14.3% on AppWorld and reaching 9.9× the strongest baseline’s score on BabyAI.
Note. This blog is an introductory version of the paper, focusing on an intuitive and empirical understanding of RSC. For theoretical proofs, please see the paper.
Reinforcement learning and off-policy data
强化学习与 off-policy 数据
Imagine asking an assistant to buy a red mug within a budget. It searches, opens a product, checks its color, and decides whether to purchase it. The sequence of decisions is a trajectory, and a reward measures how well the outcome satisfies the request. Reinforcement learning uses these outcomes to change the assistant’s future decisions. A mathematical answer can receive a correctness reward, while an interactive task can provide feedback during the trajectory as well as at its end.
At each decision, the model has a history $h$: the request, its previous actions, and any observations it has received. Its policy assigns a probability to each possible next action $a$. For a language model, the elementary action is a token. A tool call or a sentence is built from several such decisions. We write the current policy as $\pi_\theta$, where $\theta$ denotes the model’s trainable parameters.
To improve the policy, training needs to know how changing those parameters changes an action’s probability. This sensitivity is called the score:
\[\psi_\theta(h,a)=\nabla_\theta\log \pi_\theta(a\mid h).\]The symbol $\nabla_\theta$ means “take the gradient with respect to the model parameters.” A gradient tells an optimizer which parameter changes increase a quantity. Multiplying the score by a reward gives a basic policy-gradient update: reinforce choices associated with good outcomes. Here, score means a log-probability gradient, while an evaluation score measures task performance.
This update uses rewards from sampled actions, but generating that experience takes time. An agent may be waiting for a website, running a test suite, or finishing a long answer while training proceeds elsewhere. The model that started the trajectory can therefore differ from the model being updated when the reward arrives. We call the data-generating policy the behavior policy $\mu$ and the model being trained the learner $\pi_\theta$. Learning from data generated by a different policy is off-policy learning. Even freshly generated data can be off-policy if the generator uses older weights.
This distinction changes what a batch of experiences tells us. Suppose the behavior policy chooses between two actions equally often, but the learner now strongly prefers the better one. The batch still contains many examples of the other action. Giving every successful old action a positive push can pull the learner back toward the behavior policy, even when the learner already makes better choices.
One response is importance sampling (IS). It weights a sampled action by the ratio
\[\rho(h,a)=\frac{\pi_\theta(a\mid h)}{\mu(a\mid h)}.\]A ratio above one means that the learner would choose this action more often than the behavior policy. Reweighting by this ratio corrects the current-action distribution at a fixed history. However, rare behavior actions can receive very large weights, allowing a few samples to dominate an update. The history and the rest of the trajectory still come from $\mu$. These are the two issues we will separate: how to use a sampled action’s reward, and how far the learner has moved from the experience that produced it.
From IS to SC and RSC: intuition and evidence
从 IS 到 SC 与 RSC:直觉与实验
The methods below address three questions: how to use data from an older policy, how to keep a few tokens from dominating an update, and how to keep the learned policy close enough to the policy that supplied the data. We first examine one token at a fixed history $h$, since these corrections operate at the token level. The behavior policy $\mu$ stays fixed during an update; training combines the contributions across token positions.
Step 1. Policy gradient: learn from rewarded choices
第一步:Policy Gradient,根据 reward 改善选择
Policy gradient (PG) uses rewards to construct a gradient estimate. A sampled reward $R_i$ tells us how a response turned out, while its score $\psi_i=\psi_\theta(h,a_i)$ tells us how to change the selected token’s probability. Their product gives a basic contribution:
\[\widehat g_i=R_i\psi_i.\]Here $i$ identifies a sample. With on-policy trajectories, averaging these contributions and summing across their tokens gives a policy-gradient estimate. An optimizer such as SGD then uses that estimate to update the parameters. PG specifies the gradient estimate; SGD specifies how to apply it.
When responses come from an older policy, the sampling distribution changes. We next examine how IS corrects the current token’s weight.
Step 2. IS corrects the current token’s weight
第二步:IS 修正当前 token 的权重
Importance sampling (IS) compensates for the difference between how often $\mu$ selects a token and how often $\pi_\theta$ would select it. It multiplies the contribution by $\rho_i=\pi_\theta(a_i\mid h)/\mu(a_i\mid h)$:
\[\widehat g_i^{\rm IS}=\rho_iR_i\psi_i.\]In expectation, this changes the current action’s average from $\mu$ to $\pi_\theta$. It also explains why full IS needs no score centering to remove a reward baseline. A baseline has the same value for every action at a history, while the score’s mean under $\pi_\theta$ is zero:
\[\mathbb E_{\mu}[\rho\psi_\theta]=\mathbb E_{\pi_\theta}[\psi_\theta]=\nabla_\theta\sum_a \pi_\theta(a\mid h)=0.\]Multiplying this zero by the baseline still gives zero, so subtracting any such baseline leaves the expected IS update unchanged. This identity corrects the current action’s weighting; the histories and subsequent decisions in the data still follow $\mu$.
Optional derivation: why the IS weighting works
At a fixed history $h$, $Q^{\mu}(h,a)$ is the expected reward after choosing action $a$ and then following $\mu$. To see how IS changes the current action’s weighting, expand the expectation as a probability-weighted sum. The sampling probability $\mu(a\mid h)$ cancels the denominator of $\rho$:
\[\begin{aligned} \mathbb E_{\mu}[\widehat g^{\rm IS}] &=\sum_a \mu(a\mid h)\frac{\pi_\theta(a\mid h)}{\mu(a\mid h)}Q^{\mu}(h,a)\psi_\theta(h,a)\\ &=\sum_a \pi_\theta(a\mid h)Q^{\mu}(h,a)\psi_\theta(h,a)\\ &=\mathbb E_{a\sim \pi_\theta}[Q^{\mu}(h,a)\psi_\theta(h,a)]. \end{aligned}\]This requires $\mu(a\mid h)>0$ wherever $\pi_\theta(a\mid h)>0$. The final line averages the current action under $\pi_\theta$, while $Q^{\mu}$ still evaluates what happens afterward under $\mu$. Histories in the training data also remain those generated by $\mu$.
The next six tasks involve actions and environment feedback. WebShop asks the model to shop, TextWorld Cooking to prepare food, AppWorld to operate applications, BabyAI to navigate and manipulate objects, and Crafter and Craftax to gather resources and survive. Rewards measure success or progress within each task. We use agentic environments because a change in action can redirect later observations and rewards, making behavior trajectories less representative of what the learner would do.
IS corrects the current token’s weight, yet the curves remain unstable: WebShop loses part of its early gains, AppWorld’s reward declines, and BabyAI stays weak. A few tokens with very large ratios can dominate an update. This motivates limiting the multiplier.
Step 3. TIS limits the multiplier, but changes the mean update
第三步:TIS 限制乘数,也改变了期望更新
Truncated importance sampling (TIS) caps the ratio at $c$ to reduce the effect of large weights:
\[\widehat g_i^{\rm TIS}=\min(\rho_i,c)R_i\psi_i.\]This is the weighting used by CISPO. The cap reduces ratio-driven variance, but introduces bias: beyond the cap, the weights no longer turn an average under $\mu$ into an average under $\pi_\theta$. Consequently, the score’s weighted mean need not be zero, and a reward baseline can affect the expected update again.
A cap alone does not sustain learning in these experiments. TIS falls toward zero on WebShop and AppWorld, stays weak on BabyAI, and loses earlier gains on the other tasks. The curves show why we need more than a bounded importance weight; they do not isolate truncation bias as the sole cause of these failures.
We next remove the part of the update caused by giving every action the same baseline credit.
Step 4. SC removes the common reward contribution
第四步:SC 去掉 reward 中的公共贡献
At the same history, we can compare an action’s expected reward with the policy’s average expected reward over its next actions. This difference is the action’s advantage, denoted by $A$. We want the update to reflect each action’s advantage.
This comparison depends on how the policy continues after the action. When the subsequent rollout follows the behavior policy $\mu$, and the baseline also averages next actions under $\mu$, we write the advantage as $A^{\mu}$. The superscript tells us which policy supplies those future decisions and the baseline.
Score centering subtracts the mean weighted score before multiplying by the reward. Writing $\rho^{\rm TIS}=\min(\rho,c)$, SC+TIS uses
\[\widehat g_i^{\rm SC+TIS}=R_i\bigl(\rho_i^{\rm TIS}\psi_i-\mathbb E_{\mu}[\rho^{\rm TIS}\psi]\bigr).\]The mean is taken over possible next tokens at the same history.
Let $Q^{\mu}(h,a)$ be the expected reward after choosing $a$ and continuing with $\mu$, let $V^{\mu}(h)=\mathbb E_{a\sim \mu}[Q^{\mu}(h,a)]$, and let $A^{\mu}=Q^{\mu}-V^{\mu}$. At a fixed history, the mean sampled reward equals $V^{\mu}$. With the exact center,
\[\begin{aligned} \mathbb E_{\mu}[\widehat g^{\rm SC+TIS}] &=\mathbb E_{\mu}[Q^{\mu}\rho^{\rm TIS}\psi]-V^{\mu}\mathbb E_{\mu}[\rho^{\rm TIS}\psi]\\ &=\mathbb E_{\mu}[(Q^{\mu}-V^{\mu})\rho^{\rm TIS}\psi]\\ &=\mathbb E_{\mu}[A^{\mu}\rho^{\rm TIS}\psi]. \end{aligned}\]Setting every weight to one gives $\mathbb E_{\mu}[\widehat g^{\rm SC}]=\mathbb E_{\mu}[A^{\mu}\psi]$. Centering cancels the common-value term while preserving the chosen reward weights and behavior continuations.
SC therefore changes TIS’s expected update from $Q^{\mu}$-weighted to $A^{\mu}$-weighted, while $A^{\mu}$ still depends on behavior continuations. This identity describes the expected update; implementing it requires no critic or explicit estimate of $Q^{\mu}$, $V^{\mu}$, or $A^{\mu}$.
So yes, we can omit IS. The difference is which expectation we estimate. Both methods use data sampled from $\mu$. After removing the baseline contribution, SC without IS still estimates the advantage-weighted score mean under $\mu$. Full IS uses the ratio to estimate the current action’s average contribution under $\pi_\theta$ from those same behavior samples. SC+TIS retains capped weights, so it only partially corrects this weighting; the histories and continuations still come from $\mu$.
How does SC compare with the methods introduced earlier? Across the six agentic tasks below, SC+TIS sustains much higher rewards than TIS on several tasks, and both centered variants learn well on TextWorld. BabyAI remains weak, pointing to a limitation beyond the common-value contribution.
How we estimate the center also matters. Monte Carlo centering independently samples auxiliary next tokens from $\mu$ at the same history and averages their weighted scores. These tokens need no new rollouts or reward evaluations. This estimate is unbiased, whereas Top-128 approximations retain bias by using only the most likely tokens, with a modeled tail in the Countdown version.
The following comparisons use Countdown and mathematical reasoning. In Countdown, the model must form an expression that reaches a target using each of four supplied numbers exactly once; a valid answer receives reward 1 and an invalid answer 0. The mathematical reasoning tasks similarly use a verifier to check the generated answer. Every tested Monte Carlo variant finishes with a higher held-out evaluation than its corresponding Top-128 baseline. Thus accurate centering improves learning. The remaining failures motivate controlling policy mismatch.
Step 5. RSC controls the mismatch that centering leaves
第五步:RSC 限制中心化后仍存在的 mismatch
SC removes the common reward contribution, leaving an advantage-weighted update. But $A^{\mu}$ still evaluates a token using future decisions under $\mu$. Making the same comparison with rollout and baseline under the learner gives $A^{\pi_\theta}$, which can differ for the same token at the same history. This matters when a tool call changes the next observation or a token starts a solution that the two policies complete differently.
RSC adds reverse KL to encourage the learner’s next-token distribution to stay close to the behavior policy’s distribution. This limits policy movement during fitting; refreshing the behavior policy then updates the data and continuation values used by the next round.
To connect that idea to the curves, define the surrogate objective $\mathcal W_{\mu}$ as the expected sum of token contributions $w_t$:
\[\begin{aligned} \mathcal W_{\mu}(\pi_\theta)&=\mathbb E_{\mu}\!\left[\sum_t w_t(\rho_t)\right],\\ w_t(\rho_t)&=A_t^{\mu}F_a(\rho_t)-\beta(\rho_t\log\rho_t-\rho_t+1). \end{aligned}\]The first term is the reward surrogate: $F_a$ corresponds to the chosen reward weight, and becomes $\log\rho$ for SC without IS. The second term is the KL penalty, with strength $\beta\ge0$. At each history, the parenthesized expression averages to reverse KL. Thus $w_t$ is one token’s contribution, called the surrogate integrand; summing it and taking the expectation gives the full regularized objective. We hold $\mu$ and its advantages fixed when differentiating. Setting $\beta=0$ gives SC variants, while $\beta>0$ gives RSC variants.
Optional derivation: reward weights and the surrogate
Write $a(\rho)$ for the reward weight: $\rho$ for IS, $1$ for SC without IS, and $\min(\rho,c)$ for SC+TIS. Here $a$ denotes an action and $a(\cdot)$ denotes the reward-weight function. Integrating the weight divided by the ratio gives
\[F_a(\rho)=\int_1^\rho\frac{a(u)}{u}\,du,\qquad \nabla_\theta[AF_a(\rho)]=Aa(\rho)\psi_\theta.\]For IS, $F_a(\rho)=\rho-1$, so the reward-surrogate gradient is $A\rho\psi_\theta$. For SC without IS, $F_a(\rho)=\log\rho$, giving $A\psi_\theta$. For SC+TIS, it gives $A\min(\rho,c)\psi_\theta$. These match the corresponding expected updates, with advantage and behavior distribution fixed.
The KL part follows from $\mathbb E_{\mu}[\rho]=1$:
\[\mathrm{KL}(\pi_\theta\|\mu)=\sum_a \pi_\theta(a\mid h)\log\rho=\mathbb E_{\mu}[\rho\log\rho-\rho+1].\]Step 6. Read stability from the gradient curves
第六步:从梯度曲线理解稳定性
The model’s logit $z_a$ is the raw value that softmax converts into a token probability. Differentiating $w$ with respect to $z_a$ tells us how strongly the objective pushes that value. Below, $A=A^{\mu}$ is the fixed behavior advantage. The plot and table show the same derivative: the table includes the common factor $1-\pi_\theta(a\mid h)$, while the plot removes it to make the methods’ dependence on $\rho$ easier to compare.
| Method | \(\partial w/\partial z_a\) |
|---|---|
| IS | \([1-\pi_\theta(a\mid h)]A\rho\) |
| SC+TIS | \([1-\pi_\theta(a\mid h)]A\min(\rho,c)\) |
| RSC (TIS) | \([1-\pi_\theta(a\mid h)][A\min(\rho,c)-\beta\rho\log\rho]\) |
| SC (w/o IS) | \([1-\pi_\theta(a\mid h)]A\) |
| RSC (w/o IS) | \([1-\pi_\theta(a\mid h)][A-\beta\rho\log\rho]\) |
IS keeps the multiplier $\rho$; SC removes it; SC+TIS caps it. That is the main contrast in the plot. RSC also subtracts $\beta\rho\log\rho$: to the right of $\rho=1$, it opposes further probability increases; to the left, it pushes in the other direction. The plotted coefficients describe the corresponding expected updates; the empirical plots below show individual sampled gradients.
Does the same pattern appear in actual model gradients? A large ratio is possible only when $\mu(a\mid h)$ is small, since $\pi_\theta(a\mid h)\le1$. We therefore plot measured norms against the behavior probability $\mu$.
Measure both logit and parameter gradients
同时测量 logit 与参数梯度
Figure 3 compares the five update rules on the same checkpoint and token batch, holding sampled actions, response advantages, and auxiliary Monte Carlo draws fixed. The first row measures each token’s output-logit gradient norm; the second measures its gradient norm over all model parameters. SC and SC+TIS have smaller extreme gradients than IS in both spaces.
Adding KL also lowers the low-probability logit-gradient P99 for both centered variants in this batch. These measurements connect the analytical curves to actual gradients. The next section examines what happens across repeated updates, as staleness and agentic interaction make the remaining policy mismatch matter.
Failure Modes of SC and the Role of RSC in Practice
SC 在实际任务下的失败模式以及 RSC 的作用
We now bring together the results on the reasoning and interactive tasks introduced above. Learning curves show how quickly each method improves, while held-out evaluations test the resulting learner on held-out task instances.
We compare RSC with SC, SC+TIS and several importance-sampling baselines, including IS-PPOClip, which combines importance weighting with a PPO-clipped update. RSC with TIS achieves leading final performance across all six agentic tasks. Its gains are especially large on BabyAI and Crafter: compared with IS-PPOClip, the strongest baseline outside the RSC family on these tasks, its score is $9.9\times$ as high on BabyAI and 52% higher on Crafter. It also improves on the strongest established baselines by 16% on Craftax and 14.3% on AppWorld.
Why test RSC in agentic environments?
为什么使用 agentic 环境检验 RSC?
Agentic interaction can amplify off-policyness through its effect on future decisions and rewards. At the same history, $A^{\mu}$ evaluates an action through behavior continuations, while $A^{\pi_\theta}$ evaluates it through learner continuations. In an interactive task, a difference in action can change the next observation, the choices available afterward, and the eventual outcome. Those changes can widen the gap between $A^{\mu}$ and $A^{\pi_\theta}$, even at a similar sampler staleness. SC removes the common-value contribution but still uses $A^{\mu}$, so its update can favor a choice whose advantage is lower under learner continuations. This is why we use agentic tasks to test whether RSC helps when behavior-based estimates become less representative of the learner.
Figure 5 compares mathematical reasoning with Crafter at the same sampler refresh interval, $R=64$. In math, the model generates a solution and receives an answer-correctness reward. Crafter adds environment feedback between actions, letting us compare the benefit of KL across these task structures.
SC and SC+TIS sustain learning on math, while RSC with TIS learns substantially better than SC+TIS on Crafter. In an interactive environment, gathering a resource or choosing a route can open different future choices. An action’s value therefore depends on how subsequent decisions use those opportunities. This helps explain why behavior advantages can become less representative of learner outcomes. The tasks and models differ, so the next experiment varies staleness within each setting to examine policy separation more directly.
When the data fall further behind
当数据越来越跟不上模型
Task structure changes the consequences of different continuations; sampler staleness allows the policies to move further apart. We control staleness by refreshing the behavior policy only once every $R$ learner updates. Increasing $R$ lets the learner move further before the model collecting the next answers catches up. The Countdown experiment keeps the model, three-seed cohort, and training budget fixed while increasing this refresh interval.
The paper’s separate learner evaluation shows the same contrast. SC works well when refreshes are frequent, then loses most of its performance at the largest interval. RSC (TIS) retains strong held-out performance. SC+TIS also remains strong in this experiment, so the full comparison shows both the benefit of bounded importance weighting and the behavior of regularization. Unit-weight RSC improves greatly over SC at the largest interval, but retains more seed variation than RSC (TIS).
Learning curves across all six tasks
六个任务的训练对照
The training curves show how reward changes as each method collects more experience. Within each task, the methods use the same number of sampled responses per update, so higher reward at the same update count means better sample efficiency. RSC with TIS learns significantly more efficiently, with particularly clear gains on Crafter and Craftax: it stays substantially ahead of SC, SC+TIS, and IS-PPOClip through much of training. The final evaluations then check whether this advantage carries over to the trained learner.
Final evaluation: every method and task
最终评估:全部方法与任务
The table reports held-out checkpoint evaluation scores as the mean ± sample standard deviation across three seeds. WebShop measures shopping reward, TextWorld the fraction of possible cooking points, AppWorld task completion, and BabyAI step-discounted terminal reward. These scores use a percentage scale. Crafter reports its geometric achievement score, while Craftax reports total episode reward. Compare methods within each task. Bold follows the paper: the highest mean, plus the second highest if it is within 5% of the highest.
| Method | WebShop | TextWorld | AppWorld | BabyAI | Crafter | Craftax |
|---|---|---|---|---|---|---|
| IS | 40.94± 36.13 | 21.77± 16.42 | 6.43± 11.14 | 0.13± 0.13 | 1.02± 0.70 | 0.83± 0.96 |
| Seq-MIS | 47.25± 40.95 | 96.95± 3.22 | 7.02± 5.26 | 1.71± 1.75 | 1.93± 0.82 | 2.39± 1.43 |
| TIS | 0.00± 0.00 | 19.77± 13.45 | 0.00± 0.00 | 0.00± 0.00 | 0.57± 0.44 | 0.39± 0.63 |
| IS-PPOClip | 74.81± 0.28 | 99.73± 0.25 | 9.36± 3.65 | 1.83± 1.36 | 9.34± 0.46 | 4.04± 0.27 |
| SC | 41.42± 35.57 | 99.06± 0.80 | 12.28± 1.75 | 0.80± 0.95 | 1.12± 1.12 | 1.58± 1.10 |
| SC+TIS | 52.21± 30.06 | 92.14± 13.62 | 12.28± 4.64 | 0.30± 0.32 | 3.27± 1.89 | 3.20± 0.58 |
| RSC (w/o IS) | 60.46± 9.02 | 95.29± 4.32 | 11.11± 3.65 | 7.32± 5.80 | 0.94± 0.29 | 1.46± 0.64 |
| RSC (TIS) | 74.80± 1.15 | 99.95± 0.08 | 14.04± 1.75 | 18.06± 1.77 | 14.21± 3.40 | 4.71± 0.03 |
The comparison with SC+TIS is also strong: adding KL raises its WebShop evaluation score by 43%, and RSC with TIS scores close to the maximum on TextWorld. Together, the learning curves and held-out evaluations show improvements in both sample efficiency and final performance.
Fully asynchronous software engineering
完全异步的软件工程训练
In the software-engineering experiments, the model receives a task description and a code repository. It can inspect files, edit code, and run tests; a test harness checks whether its final changes solve the task and provides the success reward. We train on SWE-Gym tasks and evaluate on SWE-bench Verified. Completed trajectories enter the training queue while other agents continue editing files or running tests. The learner keeps changing during this work, so policy lag emerges naturally. We compare two configurations: A uses a learning rate of $10^{-6}$, and B uses $2\times10^{-6}$.
An RSC variant achieves the highest evaluation score in each configuration. The advantage from adding KL is much larger in B for both versions of SC: RSC without IS improves on SC by 127%, and RSC with TIS improves on SC+TIS by 51%. This stronger benefit coincides with a larger learner–behavior probability gap in the curves. It supports the central motivation: regularization matters more when behavior data become less representative of the learner.
The table reports SWE-bench Verified Average@8 (%): each task is attempted eight times, and the binary success outcomes are averaged across attempts and tasks.
| Method | A: LR $10^{-6}$ | B: LR $2\times10^{-6}$ |
|---|---|---|
| IS | 57.08 | 0.40 |
| TIS | 52.33 | 51.51 |
| SC | 58.48 | 21.13 |
| SC+TIS | 59.06 | 36.04 |
| RSC (w/o IS) | 61.14 | 47.89 |
| RSC (TIS) | 54.77 | 54.47 |
The idea to take away
最值得记住的事情
Learning from an older policy requires care in both credit assignment and policy movement. SC removes a common-value contribution from the expected update. TIS limits ratio amplification in the reward weights. KL keeps the learner’s action distribution close to the behavior policy that supplies its experience. RSC puts these pieces together in one family of updates. The resulting gains are particularly clear in interactive tasks, where many decisions separate an initial action from its outcome.
For an earlier, focused introduction to the centering mechanism, see Score Centering under Policy Mismatch. The present article accompanies our work Regularized Score Centering.