Off-Policy Policy Optimization with Proximal KL

Reward-based training for mathematical reasoning generates answers, checks them, and updates the model using correctness feedback. But how far should the model change in each round? Proximal KL balances reward against KL divergence, a measure of how probability distributions differ, relative to the round’s starting model. That starting model is the reference; “proximal” refers to penalizing movement away from it within the round. A small example will show why this can help even when the answers come from the current model, as in on-policy learning.

More importantly, generation and training often run in parallel in practice, so some answers come from older model versions: off-policy data. This raises our central question: can updates using old data preserve the optimizer, meaning the policy that maximizes this round’s KL-regularized objective? We establish conditions under which token-wise importance correction, reweighting each sampled token by its current-model probability divided by its generating-model probability, does so. A token is one unit of generated text. The key is assigning it the KL cost of the whole response that it can influence, including what is generated afterward.

Why KL can help

Why add KL when a correctness reward already identifies which responses are desirable? The choice of objective also determines the path that optimization follows. We first show a case where using the current policy as a KL reference lets optimization improve both prompts, while direct PG sacrifices one. This motivates the proximal objective. Extending it to a full response then tells us which token should receive each KL cost.

An on-policy example

Proximal KL can, in some cases, achieve a better result than ordinary policy gradient (PG), even with on-policy data. The following idealized example makes this possibility concrete by comparing the limiting reward of two optimization procedures.

A model answers different questions using the same set of weights. This is what we mean by shared parameters: an update that helps one question can also change how the model answers another. A two-question example isolates this effect.

Consider two equally likely prompts with a binary action and reward $R=a$. Use a small logistic model that maps the same parameter vector to a success probability for each prompt. It can solve both prompts, but initially succeeds with probabilities $(p_1,p_2)=(1/2,p_{\mathrm{init}})$, where $0<p_{\mathrm{init}}\le1/64$. One prompt is therefore much easier at initialization. All expectations in this example are exact and on-policy: neither stale data nor sampling noise causes the failure.

To make the model concrete, set $p_m=\sigma(u_m^\top\theta)$, where $\sigma$ is the logistic function that maps a real-valued score to a probability. This score is the logit: for a binary choice with probability $p$, it equals the log-odds $\log[p/(1-p)]$. The feature vectors describe how the same weights affect the two prompts:

\[u_1=\frac15(1,1)^\top,\qquad u_2=\frac15(1,-4)^\top,\qquad \theta_0=(-\Delta,\Delta)^\top.\]

Here Δ is the positive log-odds gap between the two initial success probabilities. Both prompts become solvable along $\theta=(c,0)^\top$ as $c\to\infty$, so this parameterization can represent joint success.

For these shared features, ordinary PG, represented by continuous-time Euclidean gradient flow on the average reward $J=(p_1+p_2)/2$, improves the easier prompt while suppressing the harder one. Gradient flow describes taking infinitesimally small steps along the exact reward gradient. Its limit is $(p_1,p_2)=(1,0)$, and $J$ never exceeds $1/2$. As correct answers to the harder prompt become rarer, that prompt contributes even less to the reward gradient. The model can solve both prompts, but this optimization path increasingly favors only one.

The key mechanism is the conflict between the two prompt gradients: $u_1^\top u_2<0$, so progress on the easier prompt can suppress the harder one. Appendix D.1 proves that this feedback persists and gives the limit above.

This failure occurs along the reward-gradient optimization path. Can a reference-dependent penalty produce a path that improves both prompts? A proximal round maximizes reward while penalizing KL deviation from the previous round’s policy, then uses the resulting policy as the next reference. Here we assume each round is solved exactly; the next subsection writes out its objective. Repeated exact KL-proximal solves in the same model class and from the same initialization give

\[(p_{K,1},p_{K,2}) =\left(\sigma(K/\beta),\sigma(-\Delta+K/\beta)\right) \longrightarrow\class{pkl-math-focus}{(1,1)}, \qquad \Delta=\log\frac{1-p_{\mathrm{init}}}{p_{\mathrm{init}}}.\]

Here $K$ counts completed proximal rounds and $\beta>0$ is the fixed KL penalty strength. Each round increases both success log-odds by $1/\beta$, then refreshes the reference. Both prompts eventually have success probability approaching one, so the average reward approaches 1, compared with the PG limit of 1/2. Proximal KL changes the optimization path: progress relative to the reference incurs a cost, reducing the easier prompt’s dominance within a round while allowing further progress after the reference refreshes.

We must also check that the original model can realize both improvements simultaneously. The parameter increment $(5,0)^\top/\beta$ realizes the exact proximal logit increase $1/\beta$ for both prompts. The proximal target is therefore representable in this same shared model at every round.

KL can improve on-policy optimization. In this example, PG approaches average reward 1/2, while Proximal KL approaches 1.

The construction and full proof appear in Appendix D.1, “Illustration of proximal feedback with shared parameters,” of our paper.

How do these updates affect success? Press Play to follow 1000 iterations from the same starting point, or drag the step slider. Compare mean reward, then switch to the harder or easier prompt’s success probability to see which question accounts for the gap between PG and repeated exact proximal solves.

This is a deterministic simulation of the two-prompt model. One PG step uses the exact reward gradient with step size $\eta=1$; one proximal round uses the analytic maximizer and refreshes the reference. These operations have different computational costs, so the counter compares optimization paths, not training speed.

Simulation update rules

PG numerically approximates the gradient flow with

\[\theta_{t+1}=\theta_t+\frac{\eta}{2}\sum_{m=1}^2 u_m p_m(1-p_m).\]

An exact proximal round increases both logits by $1/\beta$, realized here by

\[\theta_{k+1}=\theta_k+(5/\beta,0)^\top.\]

The plot reports unregularized mean reward $J=(p_1+p_2)/2$ or either prompt’s success probability after each iteration. The proximal objective changes whenever its reference is refreshed.

We next define the proximal objective and explain how its feedback changes the update.

Proximal objective

The two-prompt example in the preceding subsection repeatedly solves a reference-dependent objective. To specify what one such round optimizes, we distinguish the learner $\pi_\theta$, which changes within the round, from the reference $\pi_k$, which remains fixed. The reference is the policy from which this round begins. It defines the distribution against which we measure the cost of changing the model’s answers, even when every training response comes from the current learner.

We seek policy improvement while controlling how far the policy moves. This motivation also underlies Schulman et al.’s TRPO, which constrains KL, and PPO, which studies clipped and KL-penalized surrogate objectives. A surrogate is an auxiliary objective constructed to produce a policy update. Clipping replaces a value outside a chosen interval by its nearest bound within that objective. Here we put a learner-to-reference KL penalty on the complete response distribution.

The prompt $x$ comes from the training distribution $\mathcal D$, the response $\tau$ is a complete token sequence, and $R$ scores that response. Our objective is

\[J_k(\theta) =\mathbb E_{x\sim\mathcal D,\,\tau\sim\pi_\theta}[R(x,\tau)] \class{pkl-math-focus}{-\beta\,\mathbb E_{x\sim\mathcal D} D_{\mathrm{KL}}\!\left(\pi_\theta(\cdot\mid x)\,\Vert\,\pi_k(\cdot\mid x)\right)}.\]

The reward term favors correct answers; the reverse KL penalty controls movement away from the reference. Holding that reference fixed within a round makes the target well-defined. Refreshing it between rounds allows cumulative improvement far beyond the initial model.

Which response distribution would maximize this objective? Solving over unrestricted response distributions identifies the policy that an exact proximal round aims to reach:

\[\pi_k^*(\tau\mid x) =\frac{\pi_k(\tau\mid x)\exp(R(x,\tau)/\beta)}{Z_k(x)}, \qquad Z_k(x)=\mathbb E_{\tau\sim\pi_k(\cdot\mid x)}[e^{R(x,\tau)/\beta}].\]

For binary rewards, multiply each correct response’s reference probability by $e^{1/\beta}$, then renormalize so the probabilities sum to one. Larger $\beta$ gives a more conservative round; smaller $\beta$ amplifies correct answers more strongly. This exponentially reweighted distribution, called a Gibbs distribution, is the proximal target used below.

This solution identifies the optimization target for a round. To see how KL changes the update, return to the two-prompt example. Write the learner’s success logit as $z_m=\log[p_m/(1-p_m)]$ and the reference logit as $z_{k,m}$. The proximal objective’s gradient is

\[\nabla J_k(\theta)=\frac12\sum_{m=1}^{2} u_m p_m(1-p_m)\left[1\class{pkl-math-focus}{-\beta(z_m-z_{k,m})}\right].\]

The vectors $u_m$ describe how a parameter change affects each prompt. When their inner product is negative, improving one can hurt the other. As a prompt’s success logit increases above its reference value, the bracket reduces its reward-driven contribution; that contribution reaches zero at a logit gain of $1/\beta$. After the reference is refreshed, another round can improve both prompts further. The KL term therefore supplies an improvement target that moves with each reference refresh.

Each response in the example contains a single action. A language model generates many dependent tokens, so applying the same response-level objective raises a credit-assignment question: which costs should influence each token’s update? Using only the current token’s KL omits future-cost contributions and generally gives a biased estimate of this objective’s gradient. We now trace where those contributions come from.

Future-KL credit assignment

Why should an early token account for KL costs that occur later? Increasing its probability also makes the continuations beginning with it more likely. A continuation is the remaining sequence generated after the current choice. For example, choosing a different first step in a solution changes which later reasoning steps the model generates. Those continuations can differ in how far they move from the reference. The objective charges for the whole response, so the update for that early choice must account for its effect on later costs as well as its immediate cost.

Credit assignment means deciding which rewards and costs should influence each token’s update. Schulman et al.’s stochastic computation graphs, diagrams of dependencies among random choices and computed quantities, formalize this downstream-cost principle. Here it gives Future-KL: the sum of sampled KL log-ratio costs from the current token through the end. The computation reuses the completed response; during generation, each token is conditioned on the existing prefix, the prompt and tokens already generated.

The formula animation follows token 2 in a three-token response: remove the past term’s zero expected contribution, then retain the current and future costs.

To express this dependence in the gradient, we need a token’s cost and its sensitivity to the model parameters. Write $h_i=(x,a_{<i})$ for the prompt and all tokens before the current position. The log-ratio compares the learner’s and reference’s probabilities for the sampled token; the score vector describes how its log-probability changes with the model parameters:

\[\ell_i=\log\frac{\pi_\theta(a_i\mid h_i)}{\pi_k(a_i\mid h_i)}, \qquad \psi_i=\nabla_\theta\log\pi_\theta(a_i\mid h_i).\]

A sampled log-ratio can be positive or negative; at a fixed prefix $h_i$, its expectation over the learner’s next-token distribution gives a conditional KL divergence. Adding the token log-ratios gives the log-ratio for the complete response. Here responses come from the current learner, with a fixed reference and verifier reward, supplied by an answer-correctness checker. The paper gives the regularity assumptions for the derivation. Applying the score-function gradient identity to KL optimization gives

\[\nabla J_k =\mathbb E\left[ \left(R-\beta\sum_{j=1}^{T}\ell_j\right) \sum_{i=1}^{T}\psi_i\right].\]

This expression initially pairs every token’s score with the cost of the entire response. The optional recap below explains the differentiation step. Our next task is to determine which score–cost pairs contribute to the expected update.

Recap: differentiating the response-level KL objective

Autoregressive factorization expresses a response probability as the product of its conditional token probabilities. Taking its logarithm and differentiating turns this product into a sum of token scores:

\[\pi_\theta(\tau\mid x)=\prod_{i=1}^{T}\pi_\theta(a_i\mid h_i), \qquad \nabla_\theta\log\pi_\theta(\tau\mid x)=\sum_{i=1}^{T}\psi_i.\]

Write $J_k=\mathbb E_{\pi_\theta}[R-\beta\sum_j\ell_j]$. Differentiating the sampling probabilities gives the reward-minus-cost times the response score shown in the main text. Differentiating the learner inside the log-ratios adds $-\beta\mathbb E\sum_j\psi_j$. This second contribution has zero expectation by the zero-mean score identity established in the main discussion of past costs.

The remaining gradient therefore comes from how changing the learner changes the probability of sampling each response.

Past costs contribute zero in expectation. Condition on a prefix $h_i$ and imagine sampling its next token repeatedly. The prefix fixes everything that has already happened. Increasing some next-token probabilities must decrease others, since the probabilities still sum to one. This normalization gives a zero-mean score:

\[\mathbb E[\psi_i\mid h_i] =\sum_a\pi_\theta(a\mid h_i)\nabla_\theta\log\pi_\theta(a\mid h_i) =\nabla_\theta\sum_a\pi_\theta(a\mid h_i)=0.\]

For a past position $j<i$, the prefix contains both $h_j$ and $a_j$, so $\ell_j$ has the same value for every possible next token. Multiplying every score by this same value preserves their cancellation:

\[\mathbb E[\ell_j\psi_i\mid h_i] =\ell_j\class{pkl-math-focus}{\mathbb E[\psi_i\mid h_i]}=0, \qquad j<i.\]

This equality concerns an average over possible current actions. On an individual response, both the past cost and the score can be nonzero. The cost can also depend on the learner parameters; the direct derivative of that dependence was already included in the differentiation step. Removing the past cost from the score’s coefficient preserves the expected gradient.

Current and future costs depend on the action being updated. The current cost $\ell_i$ uses the probability of the token actually chosen. At a future position $j>i$, that choice is part of the prefix and can change the entire continuation distribution. Different next tokens can therefore lead to different expected later costs. Denote the expected cost after choosing token $a$ by $m_j(a)=\mathbb E[\ell_j\mid h_i,a_i=a]$. First average over continuations after each choice, then average over the choices themselves:

\[\mathbb E[\ell_j\psi_i\mid h_i] =\sum_a\pi_\theta(a\mid h_i)m_j(a) \nabla_\theta\log\pi_\theta(a\mid h_i).\]

The scores still have zero mean, but they are now weighted by potentially different costs. For example, a first reasoning step can lead mostly to continuations that stay close to the reference, while another leads to larger KL costs. Changing the probabilities of those first steps changes the expected later cost. The weighted sum is generally nonzero, so it belongs in the objective’s gradient. A future term can vanish in a special case, such as equal conditional costs for all choices; there is no general cancellation that lets us omit it.

A binary example of score–cost cancellation

A binary choice makes the difference explicit. Suppose the current action is $a_i\in\lbrace 0,1\rbrace$ with probability $p=\sigma(\alpha)$ of choosing 1, where $\alpha$ is its logit. The score with respect to that logit is $a_i-p$. Write $c_0$ and $c_1$ for the expected future costs after the two choices. Then this score-times-cost term is

\[\begin{aligned} \mathbb E[\ell_j(a_i-p)\mid h_i] &=(1-p)c_0(-p)+pc_1(1-p)\\ &=p(1-p)\class{pkl-math-focus}{(c_1-c_0)}. \end{aligned}\]

If both choices receive the same cost, their contributions cancel. Holding $c_0$ and $c_1$ fixed isolates the effect of changing the current choice: if choosing 1 has a higher expected future cost, raising its logit increases the expected cost. The KL penalty enters the objective with a negative sign, so this particular score–cost contribution pushes the logit downward. Shared parameters can also change the continuation costs themselves; those effects enter through the other terms of the full gradient. This calculation isolates why a zero-mean score alone does not eliminate action-dependent costs.

These two cases identify the costs for each score: remove positions before the current token and retain its complete suffix, as in the formula animation above. We may also subtract a history-only baseline, a comparison value $b(h_i)$ shared by all possible actions at the same prefix. Like a past cost, it multiplies every action’s score by the same value, so $b(h_i)\mathbb E[\psi_i\mid h_i]=0$. Holding this baseline fixed during differentiation gives the exact identity

\[\nabla_\theta J_k(\theta) =\mathbb E_{\pi_\theta}\sum_{i=1}^{T} \psi_i\left[R-b(h_i)\class{pkl-math-focus}{-\beta\sum_{j=i}^{T}\ell_j}\right].\]

The bracket is the full Future-KL advantage: reward relative to the baseline, minus the sampled KL costs attributable to the current choice. The first token receives the entire response’s cost; the last receives only its own. Keeping only $\ell_i$ drops the current action’s effect on the later distribution. The omitted terms generally have nonzero expected contributions, so the resulting gradient estimator is biased even with on-policy data. Increasing the batch size does not remove this omission bias.

This subsection establishes the gradient when responses are sampled from the current learner. The next section considers responses sampled from older policies: how does that change in sampling distribution affect the proximal target?

Stationary target

Using old responses creates a familiar trade-off. Full-response importance correction can recover the on-policy gradient, but its product of token probability ratios can have very high variance on long responses, a problem established in prior work on the curse of horizon. Token-wise ratios are consequently a common practical choice, including in the asynchronous PPO update used by AReaL. They avoid that full product, but leave the gradient generally biased. We view this gap as a potential source of performance loss: a biased update can alter where learning stops. This motivates a precise question for Proximal KL: can full Future-KL preserve the proximal optimizer as the unique stationary policy when we correct only the current token? We first define that update, then answer the question through pathwise cancellation and backward induction.

The derivation in the preceding section assumes responses sampled from the current learner. In asynchronous training, actors, the workers running response generation, operate while the learner updates and receive new weights periodically. By the time a response is used, the learner may already have changed. We call the policy that generated it the behavior policy $\pi_{\mathrm{behav}}$. Its difference from the learner makes the update off-policy.

Staleness measures the gap between the learner’s current recorded policy version and the version used to generate a response. A gap of four means weights four recorded versions behind the learner. Elapsed time and changes in response probabilities are separate measurements. In our controlled-staleness experiments, one version can contain several optimizer minibatch updates, each using a smaller portion of a collected batch. A configured cap need not equal the realized gap, so we inspect the recorded gaps.

Behavior and proximal reference serve different purposes: behavior records where data came from, while reference defines the cost in the objective. Actor synchronization and reference refresh are therefore separate decisions. Even staleness zero at the start of a batch does not make repeated updates on that batch on-policy, because the learner changes after the first update. Strict on-policy sampling uses current-policy responses for one update; its proximal reference can still remain fixed across several such batches.

Here a fixed point means a policy with zero expected update; we also call it a stationary policy. Individual sampled minibatches can still produce changes. This section identifies that policy and proves its uniqueness through Q-functions and pathwise cancellation. The following section studies whether updates approach it.

Token-wise importance correction

When behavior generates the data, some responses occur more often, and others less often, than they would under the learner. Importance sampling (IS) compensates by weighting a sampled outcome by its learner probability divided by its behavior probability. For a complete response, exact correction would use

\[W(\tau)=\frac{\pi_\theta(\tau\mid x)}{\pi_{\mathrm{behav}}(\tau\mid x)} =\class{pkl-math-focus}{\prod_{i=1}^{T}w_i}, \qquad w_i=\frac{\pi_\theta(a_i\mid h_i)}{\pi_{\mathrm{behav}}(a_i\mid h_i)}.\]

With the same prompt distribution and behavior support covering the learner’s responses, weighting the on-policy score estimator by this product restores its exact expectation. Support means the responses assigned positive probability; coverage here requires every learner-possible response to remain possible under behavior. Its weight variance can grow with response length, depending on the mismatch between the two policies. Consider independent binary tokens, with behavior probability $1/2$ and learner probability $1/2+\delta$ at every position, where $0<\lvert\delta\rvert<1/2$. Then

\[\operatorname{Var}_{\pi_{\mathrm{behav}}}[W] =(1+4\delta^2)^{\class{pkl-math-focus}{T}}-1.\]

A small discrepancy at each position can accumulate into rapidly growing importance-weight variance. The later finite-sample theorem controls this accumulation through uniform token-ratio bounds together with a small aggregate mismatch budget. Under those joint conditions, full-response weights also have bounded variance.

We instead weight each token’s update by its current-token ratio $w_i$, retaining the full, unmodified Future-KL sum. Prefixes and continuations still come from behavior, so the resulting expected update is generally biased relative to $\nabla J_k$. We call the mapping from learner parameters to this mean update vector an update field. Exact correction no longer identifies its stationary policies for us; we must determine them from this field itself.

The theoretical comparison concerns the full probability-ratio product and the current-token ratio. An ablation changes one ingredient to test its effect; its controls are the comparison configurations. In the importance-correction ablation, the sequence-level control uses the product’s geometric mean, its $T$-th root for a response of length $T$, and the IS-free control sets the weight to one. Those controls therefore use different weights from both estimators considered here.

To state the update precisely, separate the current token’s KL cost from the costs incurred after choosing it. The sampled future-only return and the complete update coefficient are

\[\widehat Q_i^{\pi_\theta} =R\class{pkl-math-focus}{-\beta\sum_{j=i+1}^{T}\ell_j}, \qquad \widehat A_i^{\mathrm{KL}} =\widehat Q_i^{\pi_\theta}\class{pkl-math-focus}{-\beta\ell_i}-b(h_i).\]

The hat marks a quantity computed from one sampled response. The superscript identifies the learner used in the KL costs; the response can still come from behavior. The sum in $\widehat Q_i^{\pi_\theta}$ starts after the current action, while $\widehat A_i^{\mathrm{KL}}$ includes the current cost as well. Separating these costs will let us determine whether behavior-generated continuations can change the stationary policy. Averaging the importance-weighted score times this coefficient defines the population update field:

\[M_{\mathrm{behav}}(\theta) =\mathbb E_{\pi_{\mathrm{behav}}} \sum_i \class{pkl-math-focus}{w_i}\psi_i \widehat A_i^{\mathrm{KL}}.\]

The essential choices are a fixed reference, positive KL strength, full unmodified Future-KL, and a detached history-only baseline. Detached means held fixed when differentiating the update. General assumptions about support, model expressiveness and sampling are specified in Appendix E.2. Here model expressiveness concerns which policies the parameterization can represent and adjust; the population update means the exact expectation of the sampled update.

Responses may come from different older versions within a batch; each retains its actual sampling probabilities. We must show both that the proximal target has zero expected update and that every zero-update policy equals that target. The next subsection proves the first statement by pathwise cancellation and the second by backward induction.

Pathwise cancellation

First, we show that the proximal optimizer is stationary for the token-wise update. Write $\pi_{\theta^\ast}=\pi_k^\ast$. Optimality gives $\nabla J_k(\theta^\ast)=0$, but our off-policy update follows $M_{\mathrm{behav}}$, which generally differs from that gradient. We must therefore prove $M_{\mathrm{behav}}(\theta^\ast)=0$ directly. The next subsection will show that no other policy has zero expected update.

The gap comes from the sampling distribution. Current-token correction changes the action expectation at a given history to the learner’s distribution, while later tokens still come from behavior. Even when the learner equals $\pi_k^\ast$, these old-policy continuations can assign different expected rewards to the current actions than continuations sampled from $\pi_k^\ast$. Their contribution to the update therefore needs to be checked explicitly. Full Future-KL provides the needed identity: at the target, reward minus all KL costs from the current token onward and the history-only baseline depends only on the current history. Current-token correction then makes the conditional mean of the score zero, so this coefficient produces zero expected update. The derivation below shows why that cancellation holds on every supported continuation.

To carry out this proof, a Q-function makes the role of each action explicit: it assigns a value to choosing token $a$ after history $h$. Here we need the soft Q-function of the proximal target, determined by the fixed reference, reward, and KL strength. “Soft” indicates that the value accounts for KL regularization. Write $h$ for the prompt and tokens already generated, and $ha$ for the new history after appending the next token $a$, not a product. For example, $h=(x,a_1)$ and $a=a_2$ give $ha=(x,a_1,a_2)=h_3$. Define

\[Z_k(h)=\mathbb E_{\pi_k(\cdot\mid h)}e^{R/\beta}, \qquad V_k(h)=\beta\log Z_k(h), \qquad Q_k(h,a)=V_k(ha).\]

The expectation in $Z_k(h)$ runs over complete continuations under the reference. Its logarithm, scaled by $\beta$, gives the soft history value $V_k(h)$. In this token-append model, choosing $a$ deterministically reaches $ha=h_{i+1}$ from $h=h_i$, so $Q_k(h,a)=V_k(ha)$. The expression $V_k(ha)$ still evaluates a state or history; $Q_k(h,a)$ names the value of choosing $a$ at $h$. At a completed response, $V_k=R$. These analytical quantities characterize the target; the training algorithm computes sampled returns directly and does not fit a Q-network, a network trained to predict action values.

The cancellation comes from a specific algebraic structure: at the proximal target, each token’s KL cost equals the difference between two adjacent soft values. Summing these differences causes the intermediate values to cancel. The final value equals the reward, so subtracting all costs strictly after the current action leaves the value immediately after that action, its Q-value. Subtracting the current token’s cost as well leaves the value before the action, $V_k(h_i)$. The animation expands this calculation for a three-token response before we derive the general identity below.

Follow the matching terms as they cancel: first the reward, then the intermediate value. No expectation is needed for this algebra; it holds along each supported path at the target.

To evaluate the KL cost at the target, first obtain its token probabilities by conditioning the response-level Gibbs distribution on a fixed prefix:

\[\pi_k^*(a\mid h)=\pi_k(a\mid h)\frac{Z_k(ha)}{Z_k(h)}.\]

The recursion $Z_k(h)=\sum_a\pi_k(a\mid h)Z_k(ha)$ normalizes the action probabilities. Taking logarithms gives the soft Q-consistency relation:

\[\beta\log\frac{\pi_k^*(a_i\mid h_i)}{\pi_k(a_i\mid h_i)} =Q_k(h_i,a_i)-V_k(h_i).\]

The current action’s KL cost is therefore its soft Q-value minus the history’s soft value. Now follow the tokens after that action. Each $Q_k(h_j,a_j)$ is the next history’s $V_k(h_{j+1})$, so intermediate values cancel and the terminal value equals $R$:

\[\beta\sum_{j=i+1}^{T} \log\frac{\pi_k^*(a_j\mid h_j)}{\pi_k(a_j\mid h_j)} =R-Q_k(h_i,a_i).\]

Substituting into the sampled return defined in the preceding subsection gives

\[\widehat Q_i^{\pi_k^*}=\class{pkl-math-focus}{Q_k(h_i,a_i)}.\]

Subtracting the current-token cost $Q_k(h_i,a_i)-V_k(h_i)$ and the history-only baseline completes the update coefficient. We can now state the resulting identity with its conditions and quantities together:

Fix reference $\pi_k$, KL strength $\beta>0$, and learner $\pi_k^\ast$. At history $h_i$, consider any continuation with positive reference probability. Let $\tau$ be the completed response and $R(\tau)$ its terminal reward. Using the complete, unclipped cost sum gives

\[\begin{aligned} \widehat A_i^{\mathrm{KL}}(\pi_k^\ast,\tau) &=R(\tau)-\beta\sum_{j=i}^{T} \log\frac{\pi_k^\ast(a_j\mid h_j)}{\pi_k(a_j\mid h_j)}-b(h_i)\\ &=\class{pkl-math-focus}{V_k(h_i)-b(h_i)}. \end{aligned}\]

Here $a_j$ is the token at position $j$, $h_j$ is its preceding history, and $T$ is the response’s final position. The sum includes the current token and every later token. $V_k(h_i)$ is the optimal KL-regularized expected return from history $h_i$ with reference $\pi_k$; $b(h_i)$ is a baseline fixed by that history.

Different continuations can have different rewards and KL costs, but their difference in this coefficient is always the same $V_k(h_i)-b(h_i)$. The equality holds for each supported response individually, including one generated by an older behavior policy. Changing which continuation is sampled cannot change this coefficient, whose common value can be nonzero. This is pathwise cancellation:

At the proximal target, reward minus full Future-KL equals the same history value on every continuation.

This cancellation gives stationarity after averaging over the current action. Condition on the current history: the token importance weight converts the action expectation from behavior to the target policy, whose score has mean zero. The coefficient is the same for every action, so its product with that mean score is zero. This proves that the proximal target has zero expected update.

Uniqueness by backward induction

Why is this the only stationary policy? The proof reduces the multi-step decision problem to one KL-regularized bandit at each history. Its payoff is the value assigned to choosing an action: here, the expected terminal reward minus the KL costs of later actions. Once later-token policies equal the proximal target, pathwise cancellation makes this payoff equal the current action’s exact Q-value. The bandit’s unique solution then fixes the current policy. Working backward from the final token fixes the whole policy.

First collect the continuation’s contribution into an action value, including when the learner is away from the target. For a single behavior policy, average the sampled future-only return after fixing the history and current action:

\[\widetilde Q_{\mathrm{behav}}(h,a) =\mathbb E_{\pi_{\mathrm{behav}}} \left[\widehat Q_i^{\pi_\theta}\mid h_i=h,a_i=a\right].\]

The remaining tokens follow behavior, while the KL costs inside $\widehat Q_i^{\pi_\theta}$ use the learner and reference. Consequently, this conditional Q-value can depend on behavior and differ from the target’s $Q_k$.

For multiple behavior versions chosen before response generation, current-token correction gives a mixture of these Q-values weighted by the version probabilities conditional on the prefix. Each response must retain its actual sampling probabilities; Appendix E.2 derives the mixture.

At each history, this is a KL-regularized bandit update. For this local calculation, fix behavior and the learner’s policies at all later histories. Each conditional Q-value is then a fixed payoff for choosing its action. Use independently adjustable action logits, scores converted to probabilities by exponentiating and normalizing. Write $p(a)=\pi_\theta(a\mid h)$ and optimize

\[\max_{p\in\Delta(\mathcal A(h))} \left\{ \sum_a p(a)\class{pkl-math-focus}{\widetilde Q_{\mathrm{behav}}(h,a)} -\beta\sum_a p(a)\log\frac{p(a)}{\pi_k(a\mid h)} \right\}.\]

Here $p$ ranges over distributions on the available actions. The objective trades expected payoff against KL from the reference. Its local-logit gradient gives the conditional token-wise update after importance correction; positive history occupancy, the probability of visiting that history, only rescales it. The baseline and the constant $-\beta$ from differentiating KL vanish by the zero-mean score identity. With $\beta>0$, the objective is strictly concave in $p$, so it has only one maximizing action distribution. Under full support, meaning positive probability for every allowed action, zero local update identifies that normalized optimum:

\[\pi_\theta(a\mid h) =\frac{\pi_k(a\mid h) \exp\!\left(\widetilde Q_{\mathrm{behav}}(h,a)/\beta\right)} {\sum_{a'}\pi_k(a'\mid h) \exp\!\left(\widetilde Q_{\mathrm{behav}}(h,a')/\beta\right)}.\]

Across the tree, the bandit’s payoff $\widetilde Q_{\mathrm{behav}}$ depends on the learner’s later-token policies, so we determine those payoffs from the end backward. At the final token, the future-only sum is empty, so $\widehat Q_T^{\pi_\theta}=R$ and $\widetilde Q_{\mathrm{behav}}=Q_k$. The payoff is simply the terminal reward, and the normalized policy above is exactly $\pi_k^\ast$.

Suppose every later conditional policy already equals the target. Their KL costs telescope on every sampled suffix, so $\widehat Q_i^{\pi_\theta}=Q_k(h_i,a_i)$ and hence $\widetilde Q_{\mathrm{behav}}=Q_k$ at the current history, regardless of which covered behavior version generated the suffix. This identity uses only the later policies: it fixes the bandit’s payoff while leaving the current policy free to vary. The unique bandit optimum therefore forces the current policy to equal $\pi_k^\ast$. Repeating this argument backward establishes uniqueness across the entire tree.

Appendix E.2 gives the expressiveness conditions that extend this history-by-history argument to shared parameters.

Both directions are now established. Pathwise cancellation gives zero expected update at the proximal optimizer; the local bandit condition and backward induction exclude every other stationary policy. In the setting of Appendix E.2,

\[M_{\mathrm{behav}}(\theta)=0 \quad\Longleftrightarrow\quad \class{pkl-math-focus}{\pi_\theta=\pi_k^\ast}.\]

Full-response correction recovers the exact on-policy proximal gradient. Taking behavior equal to the learner in the same argument also gives this gradient the unique stationary policy $\pi_k^\ast$. We can therefore conclude the comparison posed earlier:

Token-wise and full-response correction share one unique stationary policy: the proximal optimizer.

The proof uses the full, unmodified future sum and a detached history-only baseline. Altering the cost sum can prevent it from telescoping to a history-only quantity, so the target may cease to be stationary. The practical update subsections explain the effects of horizon truncation, cost clipping, and group-relative normalization separately.

Appendix E.2 of our paper, “Token-Wise Importance Weighting and Its Stationary Target,” gives the full assumptions, pathwise identity, and uniqueness proof.

Connection to Q-learning

Why does a proof about policy updates keep returning to Q-functions? Both the preceding argument and Q-learning describe a current action through the value of what follows it. Our Q-learning discussion explains how a Bellman backup propagates that value backward. Here the same structure appears with KL regularization: substituting $Z_k(ha)=\exp(Q_k(h,a)/\beta)$ into the recursion for $Z_k(h)$ gives

\[V_k(h)=\beta\log\sum_a\class{pkl-math-focus}{\pi_k(a\mid h)} \exp\!\left(\frac{Q_k(h,a)}{\beta}\right).\]

Together with $Q_k(h,a)=V_k(ha)$ and terminal value $R$, this is a reference-weighted soft Bellman recursion. Its log-sum-exp exponentiates the scaled action values $Q_k/\beta$, sums them with reference weights, then takes the logarithm. The resulting reference-weighted Boltzmann probabilities are the normalized exponential weights derived above. This is the structure behind soft Q-learning, specialized to our fixed-reference objective. At the target, the policy’s log-ratio encodes the soft advantage, action value minus history value: $\beta\log(\pi_k^\ast/\pi_k)=Q_k-V_k$.

The learned quantity and update are different. Soft Q-learning fits action values using targets built from later value estimates. Our update learns $\pi_\theta$ directly from sampled reward-minus-Future-KL, weighted by current-token IS; $Q_k$ is used in the proof, with no fitted Q-network. Schulman, Chen and Abbeel make the broader connection precise: under their parameterization and sampling conditions, a soft Q-learning loss gradient decomposes into a regularized policy-gradient term and a value-fitting term. This explains the shared soft-advantage structure without identifying the two training rules.

Path consistency is the closest connection. It requires policy probabilities and values to agree with the rewards and regularization costs accumulated along a path. PCL and Trust-PCL use soft consistency identities across on-policy and off-policy paths to train by reducing consistency errors. We use this mechanism to show that a sampled future return becomes the exact target Q-value, then establish the stationary policy of the importance-weighted score update. The path-by-path equality uses deterministic prefix transitions and fixed terminal rewards on our generation tree. With stochastic environment transitions, a Bellman relation generally averages over the next state.

The soft Q identities establish where our update vanishes. We next study whether the actual policy updates approach that target.

Convergence

For an update to learn, it must approach the target from other policies. A unique stationary policy alone does not establish this. We first use a simple example to show the gap, then analyze convergence with a fixed reference, improvement across reference refreshes, and the reward attainable with finitely many sampled responses.

Stationary points and update dynamics

A simple counterexample shows why the target alone does not determine optimization speed. We can either ascend a reward objective or minimize its squared gradient: both can reach a zero-gradient solution, but their update rules differ. For the one-dimensional objective $J(\theta)=-\theta^4/4$, the two continuous-time updates are

\[\begin{aligned} \nabla J(\theta)&=-\theta^3, &\dot\theta&=-\theta^3,\\ \mathcal E(\theta)&=\tfrac12\lvert\nabla J(\theta)\rvert^2=\tfrac12\theta^6, &\dot\theta&=-3\theta^5. \end{aligned}\]

Here $\dot\theta$ denotes the rate of parameter change in continuous time. Both dynamics have the same unique stationary point, $\theta=0$. From a nonzero initialization, their distance to zero decays at rates $t^{-1/2}$ and $t^{-1/4}$, respectively. Squaring the gradient preserves the desired solution here but slows the approach near it. For our token-wise update, we therefore analyze the update field away from the target. The Q-functions introduced in the previous section let us express its remaining error through the learner’s later-token policies.

Convergence to the target

Keep the reference fixed and follow the policy before it reaches the target. At the target, pathwise cancellation gives an exact Q-value on every supported continuation. Away from it, subtracting that same target identity from the learner’s sampled future-only return isolates the error:

\[\widehat Q_i^{\pi_\theta}-Q_k(h_i,a_i) =\class{pkl-math-focus}{-\beta\sum_{j=i+1}^{T} \log\frac{\pi_\theta(a_j\mid h_j)}{\pi_k^*(a_j\mid h_j)}}.\]

Only later-token policy errors appear on the right. At the final token, this error is already zero because the future-only sum is empty. As later policies approach their targets, the perturbation of the current action’s Q-return vanishes. This gives a backward route through the generation tree: establish progress for the final decisions, then control how their remaining errors affect earlier decisions.

The convergence analysis follows exact expected updates in continuous time, with independently adjustable token logits and continued exploration of each history. It uses full Future-KL and a history-only baseline. Appendix E.3 specifies the technical conditions, including those permitting behavior to change over time.

The continuous-time token-wise population update converges to the proximal target.

Appendix E.3 of our paper, “Population Convergence and Outer Improvement,” gives the complete dynamics argument. The finite-sample result below separately specifies a sampled update rule and bounds its final reward.

Improvement across reference refreshes

Why would reaching each round’s target improve reward across rounds? Start with the exact Gibbs solution for a fixed prompt whose reference success probability is $p$. Under binary rewards, its updated success probability is

\[p^+=\frac{pe^{1/\beta}}{1-p+pe^{1/\beta}}.\]

For $0<p<1$, we have $p^+>p$. In log-odds,

\[\log\frac{p^+}{1-p^+} =\log\frac{p}{1-p}+\class{pkl-math-focus}{\frac1\beta}.\]

Each exact proximal round increases the success log-odds by the same amount. Refreshing the reference lets these gains accumulate across rounds.

Finite-sample reward guarantee

The exact recurrence leaves a practical question: what reward can finitely many sampled updates achieve? For binary rewards, our finite-sample theorem answers this for the projected linear-logit update analyzed in Appendix F. Linear logits are weighted sums of fixed features; projection keeps the parameters inside a prescribed region after each update.

The update uses current-token IS, full Future-KL and a group-centered reward-minus-KL residual: compute reward minus KL cost for each response, then subtract the group’s mean of that quantity. Appendix F gives its model assumptions, fresh-group sampling protocol and reference-selection rule.

The reward guarantee also requires a precise measure of how far behavior differs from the learner. At each history, take the learner-weighted second moment of the token log-ratio, its expected square. Sum these conditional moments along a response path, and require a uniform upper bound across supported paths and all training updates:

\[\sum_{i=1}^{T} \mathbb E_{a\sim\pi_\theta(\cdot\mid h_i)} \left[\left(\log\frac{\pi_\theta(a\mid h_i)} {\pi_{\mathrm{behav}}(a\mid h_i)}\right)^2\right] \le V_{\log}.\]

This sums learner-weighted log-ratio second moments along a path. The bound must hold for every supported path throughout training, including policy randomness.

One sufficient case uses $1/2\le w_i\le2$ and $V_{\log}\le1/8$, alongside the technical conditions and training budget in Appendix F.3. The log-ratio budget is zero on policy because learner and behavior coincide.

With these conditions in place, write the policy’s prompt-averaged success probability as $J(\pi)=\mathbb E_x\Pr_\pi(R=1\mid x)$. We compare the trained policy with a comparator $\pi_\star$ whose per-prompt correct-response probability is at most $C_\star$ times the initial policy’s. The theorem gives

\[\mathbb E[J(\pi_K)]\ge \class{pkl-math-focus}{J(\pi_\star)-\varepsilon}.\]

The theorem supplies a polynomial scored-response budget, specified in Appendix F.3. The guarantee concerns final single-response success, averaged over training randomness and reference selection.

Oracle best-of-$N$ gives this comparator condition a concrete interpretation: sample $N$ initial-policy responses and select a correct one whenever available. If initial single-response success is $p_0(x)$,

\[1-(1-p_0(x))^N\le Np_0(x).\]

Thus $C_\star=N$ covers that comparator. Under the theorem’s conditions, a single trained-policy response can approach the reward of initial-policy oracle best-of-$N$. The oracle comparator uses answer correctness to select its output. The training budget counts the scored responses used to learn the final policy.

Appendix F specifies the analyzed update and sampling assumptions; Appendix F.3, “Complete finite-sample reward guarantee,” states and proves the reward bound in our paper.

Experimental staleness counts policy versions. The same version gap can produce small probability changes during slow learning or large ones during a sharp update. Establishing the theorem’s mismatch condition therefore requires control of the token log-ratios themselves; a logged lag value alone cannot establish it.

Initial correct-solution coverage matters: probability reweighting cannot create support for a correct response whose initial probability is zero. The comparator factor and representation assumptions express this limitation explicitly.

Practical updates

To train with Future-KL, we need an update computable from sampled responses: which probabilities to record, which costs to sum, and when to refresh the reference. We specify this algorithm first, then show how horizon, cost bounds, and group normalization change each token’s update coefficient.

Proximal KL algorithm

Each Proximal KL update records the sampling probabilities, computes reward advantage and Future-KL, then updates the learner. The animation follows the algorithm one line at a time, with the full procedure kept in view. Training uses sampled rewards and token probabilities; it does not fit the Q-function used in the proof.

The reference defines the KL cost; recorded behavior probabilities define the sampling correction. They may come from the same policy, but their roles remain separate.

For position $i$ in response $m$, constructing the update coefficient starts with its KL cost. The practical version allows a finite credit horizon, the number of positions whose costs enter this token’s coefficient, and per-token cost clipping, giving

\[C_{m,i}^{(H)} =\beta\sum_{j=i}^{\class{pkl-math-focus}{\min(T_m,i+H-1)}} \operatorname{clip}\!\left( \log\frac{\pi_\theta(a_{m,j}\mid h_{m,j})}{\pi_k(a_{m,j}\mid h_{m,j})}, \class{pkl-math-focus}{-\epsilon_F,\epsilon_F}\right).\]

The window includes the current token and at most $H-1$ later tokens. Full tail extends to the response endpoint; $\epsilon_F=\infty$ removes per-cost clipping. Costs accumulate as an unnormalized sum over the window.

The coefficient also needs a reward signal. Comparing a response’s reward with those of other responses to the same prompt supplies the response-level advantage $\widehat A_m$, a positive or negative coefficient shared by that response’s tokens. Each position subtracts its own $C_{m,i}^{(H)}$ and applies the current-token importance weight. To turn this coefficient into a parameter update, one implementation minimizes the detached score loss

\[\mathcal L^{(H)}(\theta) =-\frac{1}{N_{\mathrm{tok}}}\sum_{m,i} \operatorname{sg}\!\left[ w_{m,i}\bigl(\widehat A_m\class{pkl-math-focus}{-C_{m,i}^{(H)}}\bigr)\right] \log\pi_\theta(a_{m,i}\mid h_{m,i}).\]

Here $N_{\mathrm{tok}}$ counts non-padding action tokens in the minibatch. Stop-gradient, $\operatorname{sg}$, treats the coefficient as fixed during differentiation; the next update recomputes it. The loss gradient has a minus sign, so gradient descent adds the token score multiplied by its detached, importance-corrected coefficient, averaged over these tokens. When response lengths vary, the sampled token-count denominator can also change the expected field relative to the response-averaged population update analyzed earlier.

Horizon and cost bounds

A smaller $H$ removes terms from the full Future-KL coefficient and generally changes its expected score update. Even with on-policy sampling, truncation introduces bias relative to the complete gradient whenever the omitted terms’ combined expected score contribution is nonzero. Under token-wise IS, this truncation acts on a full-tail field that already has sampling bias.

Future costs are random and can be correlated with one another, so removing terms can change sampling variance in either direction. A bias–variance trade-off must account for both effects: bias describes how far the expected update departs from the desired update, while variance describes fluctuations caused by sampling. Variance reduction and improved training performance are possibilities to measure in a particular setting.

The cost bound $\epsilon_F$ limits each log-ratio before it enters the earlier tokens’ future sums. Clipping changes the coefficient and can bias its expected score contribution. The resulting magnitude bound describes this altered coefficient; its sampling variance also depends on correlations among terms. For finite $H$ and $\epsilon_F$,

\[|C_{m,i}^{(H)}| \le\beta\epsilon_F\min(H,T_m-i+1).\]

The controls act on different aspects of the update. $H$ selects which positions participate; $\epsilon_F$ bounds each contribution; $\beta$ rescales the feedback. Even equal $\beta H$ can assign credit to very different tokens.

A finite horizon shortens the range of KL credit assigned to the current score. The model still sees the full prefix, and generation uses the same response limit. Our tail-decay variant keeps unit weights for the first $H$ costs and linearly downweights later costs to zero at the endpoint. This reweighting generally changes the expected update; the net change depends on the affected terms’ combined expected score contributions.

Different Algorithms

Start from PG with token-wise importance correction. One branch adds PPO clipping to obtain the GRPO comparator; the other subtracts Future-KL to obtain Proximal KL. The animation keeps the shared loss terms in place and highlights each branch’s change. Select a branch to follow its algorithm.

The experiments compare this additive cost with PPO clipping, which acts on a different part of the update. Given the sign of an advantage, PPO clipping gates further changes in a proximal ratio $\rho_i=\pi_\theta/\pi_k$. Above the upper bound, a positive-advantage contribution can be switched off while a negative one remains. Proximal KL supplies additive feedback: positive future log-ratios reduce the coefficient, while negative costs can increase it. At the cost bound, the clipped contribution remains part of the token’s score coefficient.

The update differences can be made explicit with surrogate losses, differentiable expressions constructed to produce the desired update direction. All methods use the same verifier reward; their surrogates produce different update fields. We write ascent surrogates below, with common token normalization omitted; implementation minimizes their negatives. PG uses reward advantage and the current-to-behavior weight:

\[\mathcal L_{\mathrm{PG}} =\mathbb E_{\pi_{\mathrm{behav}}}\sum_i w_i\operatorname{sg}[\widehat A_m].\]

Differentiating $w_i$ produces $w_i\nabla\log\pi_\theta$. For the same detached coefficient, ascent in this ratio form gives the same update as descent in the negative score loss above: PG uses $\widehat A_m$, and Proximal KL uses $\widehat A_m-C_i^{(H)}$. An implementation chooses one differentiable path; differentiating both the weight and a score-loss factor would add an unintended term.

The GRPO comparator clips the proximal ratio and multiplies by fixed reference-to-behavior correction:

\[\mathcal L_{\mathrm{GRPO}} =\mathbb E_{\pi_{\mathrm{behav}}}\sum_i \frac{\pi_k}{\pi_{\mathrm{behav}}} \min\!\left( \rho_i\widehat A_m, \class{pkl-math-focus}{\operatorname{clip}(\rho_i,1-\epsilon,1+\epsilon)}\widehat A_m \right).\]

CISPO clips the detached learner-to-behavior weight while retaining the score contribution:

\[\mathcal L_{\mathrm{CISPO}} =\mathbb E_{\pi_{\mathrm{behav}}}\sum_i \operatorname{sg}\!\left[ \class{pkl-math-focus}{\operatorname{clip}(w_i,c_{\mathrm{lo}},c_{\mathrm{hi}})}\widehat A_m \right]\log\pi_\theta(a_i\mid h_i).\]

Proximal KL adds future credit to the token coefficient:

\[\mathcal L_{\mathrm{PKL}} =\mathbb E_{\pi_{\mathrm{behav}}}\sum_i w_i\operatorname{sg}[\widehat A_m\class{pkl-math-focus}{-C_i^{(H)}}].\]

Compared with PG, it changes regularization credit; compared with GRPO, it changes how movement is controlled; compared with CISPO, it explicitly includes future cost relative to a fixed reference. The experiments can therefore ask separately whether correction alone suffices, whether bounding weights suffices, and how clipping compares with additive future feedback.

The GRPO variant here omits the initial-reference KL penalty. A permanent penalty toward initialization and a local reference refreshed between rounds define different objectives; Proximal KL uses the latter.

Reference refresh

The surrogate formulas describe an update with a fixed reference. To specify a training run, we must also decide when to refresh that reference. We use $L$ for optimizer updates within a round and $K$ for the total number of rounds. Within a round, $\pi_k$ is fixed, behavior synchronizes on its own schedule, and the learner keeps updating. The zero-lag Proximal KL comparisons refresh reference every 50 updates. In ordinary off-policy minibatch comparisons, reference probabilities are recomputed at the start of a global batch and retained across its minibatches. Both fit the framework; each experiment must still be read with its own configuration.

Alongside reference timing, the score loss requires a concrete rule for its group-relative advantage. With a verifier that supplies only final correctness, multiple responses to the same prompt provide a comparison: subtracting the group mean gives higher coefficients to responses that outperform their group. The periodic-refresh experiment uses this centered reward; controlled-staleness experiments also divide it by the group standard deviation.

The group mean and standard deviation are computed from sampled responses, including the response being updated. They depend on sampled actions and rewards, while the baseline in the fixed-point proof is fixed by the current history. Dividing the reward coefficient by the group standard deviation also changes its scale relative to the separately computed KL cost, giving groups with different reward spreads different reward-to-KL balances. The finite-sample theorem analyzes a specified group-centered reward-minus-KL residual; the practical update centers and normalizes rewards before subtracting the separately computed KL cost.

The update gives each ablation a concrete question: where should sampling correction act, how does baseline error enter, how large should each cost be, and how far into the future should it reach?

Main results

Does the resulting update sustain useful learning when rollout policies, the policies generating responses, fall behind, and do its gains extend to new problems? We test both questions on language-model mathematical reasoning. Stability tests track learning through actor refreshes and controlled staleness; independent checkpoint evaluations compare accuracy across eight held-out benchmarks. A checkpoint is a saved model-weight snapshot. This order connects the behavior during training to the performance of the resulting models.

Our working default is $H=\mathrm{full}$, $\beta=0.30$, and $\epsilon_F=0.05$. The periodic-refresh experiments use this same choice at all four model sizes. A shorter horizon offers a simple refinement: for 4B, meaning four billion model parameters, we suggest trying $H=32$ with the same coefficient and cost bound. This finite-window suggestion is a starting point for subsequent runs. The reported 4B checkpoint results use the configurations documented in the settings note below.

Reported configurations and hyperparameter selection

We selected the final Proximal KL configurations through a few simple manual adjustments under a limited tuning budget. We inspected early learning curves and adjusted hyperparameters when improvement was insufficient, then retained the resulting configurations for stable training. The mechanism ablations separately examine particular update choices. Appendix C.2 of the paper describes this selection process.

Periodic refresh uses Slime on DAPO-Math-17K, learning rate $10^{-6}$, and batches of 32 prompts with eight responses each. Each fresh batch receives one optimizer update. Rollout weights refresh every 64 updates; Proximal KL’s reference refreshes every 16 and GRPO’s every 32. The 1.7B, 4B, and 8B models use base checkpoints, taken after pretraining. The 32B model uses a post-trained checkpoint, further trained after pretraining, because the corresponding pretrained checkpoint was unavailable. Prompt and response limits are 1,024 and 16,384 tokens; temperature is 1.0. Temperature controls the sharpness of token sampling probabilities; 1.0 leaves logits at their original scale. Appendix B.2 gives the full settings.

Controlled-staleness training uses AReaL, Qwen3-1.7B-Base and Qwen3-4B-Base, and 9,453 filtered math prompts. Each global batch has 128 prompts with 16 responses each, an 8,192-token response limit, and temperature 1.0. Off-policy batches receive four optimizer minibatch updates. At nonzero staleness, the reference is recomputed before each global batch and held fixed for its minibatches. Adam uses a constant learning rate and no weight decay, with gradient-norm clipping at 1: it rescales the whole gradient when the vector’s length exceeds 1. All compared 4B methods use dynamic sampling: prompt groups with identical rewards are replaced to fill the batch.

The reported nonzero-staleness Proximal KL runs all use $\epsilon_F=0.05$:

1.7B, learning rate $5\times10^{-6}$, staleness 4 and 16: full tail, $\beta=0.30$.
1.7B, learning rate $10^{-5}$, staleness 4: full tail, $\beta=0.30$.
1.7B, learning rate $10^{-5}$, staleness 16: $H=64$, $\beta=0.60$.
4B, learning rate $5\times10^{-6}$, staleness 4: the first 32 tokens have unit weight, followed by a linearly decayed tail; $\beta=0.25$.
4B, learning rate $5\times10^{-6}$, staleness 16: $H=8$, $\beta=0.25$.

The separate zero-staleness comparisons use either one or four optimizer minibatches. Their Proximal KL runs use full tail, $\beta=0.03$, unbounded future costs, and a reference refreshed every 50 updates. The one-minibatch controls omit PPO clipping. These run settings are recorded in Appendix C.2.

Stability tests

We first make the mismatch easy to see: each iteration collects a fresh response batch and uses it once for a single optimizer update, while rollout weights synchronize only every 64 updates. Between synchronizations, the learner keeps moving while the actor generates new responses from the same older policy. This tests whether the different updates sustain progress as those two policies separate.

This experiment uses Slime and compares PG, CISPO, GRPO, and Proximal KL at a learning rate of $10^{-6}$. Proximal KL uses the full-tail working default at 1.7B, 4B, 8B, and 32B. Rollouts use FP8, an 8-bit floating-point format, for weights and activations. Their attention key/value cache, stored computations reused during generation, uses the 16-bit BF16 format; optimization also uses BF16. Training accuracy is the fraction of responses accepted by the verifier. Group rewards are mean-centered without standard-deviation normalization, and the curves use EMA $\alpha=0.10$.

Online curves are EMA-smoothed; aggregation across seeds is reported separately.

Periodic actor refresh across four model sizes. Each fresh batch receives one update; actor weights refresh every 64 updates. All Proximal KL curves use the full-tail working default. Training accuracy uses EMA 0.10.

The second half of training reveals whether early gains persist. CISPO makes substantial initial progress at 1.7B before its accuracy falls sharply. At 8B and 32B, PG and CISPO also undergo large training-accuracy declines. Proximal KL retains its gains across all four scales with the same working default.

All four methods use importance weighting, and their observed stability depends on how they control policy movement. PG follows reward, CISPO bounds the weights, GRPO uses PPO clipping, and Proximal KL subtracts a future KL cost from the learning signal.

GRPO also sustains training, and Proximal KL reaches higher accuracy with fewer updates in this periodic-refresh experiment. At 8B, its smoothed training accuracy first reaches roughly 50% after about 200 updates, compared with about 640 for GRPO. Proximal KL also has higher mean raw training accuracy over the final 100 updates shared by the two methods at 1.7B, 4B, and 8B, showing that the gain persists beyond the initial rise.

Periodic refresh lets the actor fall progressively behind between synchronizations. To compare methods at specified rollout ages, we also run controlled-staleness experiments. Within each method comparison, model scale, prompt source, learning rate, sampling, and evaluation protocol are matched. The controlled-staleness experiments use Qwen3-Base and 9,453 filtered math prompts. Adam uses a constant learning rate and no weight decay. Ordinary off-policy batches are reused across four optimizer minibatches; GRPO uses symmetric PPO clipping with width 0.2.

We train Qwen3-1.7B and 4B in AReaL at controlled rollout staleness 4 and 16, with generation overlapping optimization. The 1.7B comparisons use learning rates $5\times10^{-6}$ and $10^{-5}$; 4B uses $5\times10^{-6}$. We verify realized policy-version gaps in the logs. For the 1.7B GRPO comparison at staleness 4, the recorded mean is approximately 4.

The two protocols test different uses of old-policy data. Periodic refresh uses each fresh batch once while keeping actor weights fixed between synchronizations. Controlled-staleness training combines overlapping generation with four minibatch updates per batch, so both rollout delay and response reuse contribute to the mismatch. It standardizes rewards within each prompt group and refreshes the proximal reference before each global batch.

Before reading the held-out curves, it helps to distinguish the evaluation metrics. MATH500, OlympiadBench, MinervaMath, and Gaokao2023MathEn use greedy pass@1: one response per problem, selecting the highest-probability next token at each step. Each AIME24/25 and AMC23/24 problem receives 16 sampled responses, generated by drawing tokens from the configured sampling distribution.

Mean@16 asks how many generated responses are correct. Pass@16 asks whether a problem was solved at least once. Solving a problem once in 16 attempts already counts as success for pass@16, while its mean@16 is only $1/16$. A method can therefore improve coverage without improving single-response reliability to the same extent. It can also become more reliable on problems it already solves without covering many new ones.

The connection between these metrics comes from the chance of missing the problem on every attempt. In an idealized model with independent attempts, each succeeding with probability $p$, subtracting that all-failure probability from one gives

\[\mathbb E[\mathrm{pass@16}\mid p]=1-(1-p)^{16}.\]

The relationship is nonlinear: an increase in success probability adds little coverage for an already reliably solved problem, but can add much more for a rarely solved one. Problems have different success probabilities, so substituting aggregate mean@16 into this formula does not recover the actual pass@16. To obtain expected coverage under this model, the formula must be applied to each problem’s success probability before averaging across problems.

Accuracy tells us whether answers remain correct; we also want to know how the policy distribution changes as training proceeds. We therefore plot AIME24 pass@16 alongside entropy across six settings. Lower entropy means the next-token distribution concentrates on fewer choices; higher entropy means its probability mass is more dispersed. Both correct and incorrect answers can be generated confidently, so entropy alone cannot establish reasoning quality. Reading it together with held-out accuracy shows whether changes in the distribution accompany sustained gains or declining performance.

Accuracy and entropy provide complementary views of stability. Six settings vary model scale, learning rate, and rollout staleness.

For the smaller model at the lower learning rate, PG can be competitive. That result does not reliably carry over to larger models or more aggressive updates. At 4B and staleness 16, PG shows a pronounced AIME24 decline and entropy fluctuations; higher-learning-rate PG also fails to consistently retain its early gains. GRPO and Proximal KL sustain more stable trajectories, and Proximal KL reaches higher late AIME24 pass@16 in all four $5\times10^{-6}$ settings. We therefore focus the benchmark comparison on these two methods. The GRPO clipping-width ablation examines another source of instability: widening its clipping interval produces larger gradient excursions and a temporary accuracy decline.

Held-out evaluation

The training-stability comparison tests methods across increasingly difficult update settings. To examine accuracy more closely at $5\times10^{-6}$, we compare both mean@16 and pass@16 across the four settings, covering response accuracy and problem coverage.

AIME24 at learning rate 5e-6 across four matched settings. Mean@16 measures response accuracy; pass@16 measures problem coverage. Curves use EMA 0.20.

Proximal KL has higher late-training AIME24 mean@16 and pass@16 in all four settings, improving both response accuracy and problem coverage. We next compare fixed checkpoints across a broader benchmark suite to assess how far these gains extend.

The stale-data results also motivate a control question: how do the methods compare when every update uses freshly sampled responses? The checkpoint comparison therefore includes an on-policy setting that deliberately changes how data are reused: each fresh batch receives one update, and the GRPO control disables PPO clipping. Proximal KL uses the full tail, $\beta=0.03$, no future-cost clipping, and a reference refreshed every 50 updates. This comparison jointly specifies sampling staleness, data reuse, and the update rule, so its result applies to that complete protocol.

“lag=0” describes the sampling version relative to the learner at the start of processing a batch. Four consecutive updates reuse the answers generated before the first update: their behavior probabilities stay fixed while the learner changes. The later minibatch updates therefore operate on data from an older policy. The table reports one-update on-policy training and zero-lag, multiple-minibatch off-policy training as separate settings.

We evaluate fixed checkpoints on eight benchmarks. MATH500, OlympiadBench, MinervaMath, and Gaokao2023MathEn use greedy pass@1 on 500, 674, 272, and 383 problems. AIME24, AIME25, AMC23, and AMC24 use mean@16 on 30, 30, 40, and 45 problems, respectively. The table’s Average is micro accuracy: it pools correct responses across these eight benchmarks, with 500, 674, 272, 383, 480, 480, 640, and 720 responses, totaling 4,149 per seed. Each response contributes once. Pass@16 describes problem coverage and is reported separately in the learning curves.

\[\mathrm{Average} =100\,\frac{\sum_{d\in\mathcal D}N_{\mathrm{correct},d}} {\class{pkl-math-focus}{\sum_{d\in\mathcal D}N_{\mathrm{responses},d}}}, \qquad \sum_{d\in\mathcal D}N_{\mathrm{responses},d}=4149.\]

Here $\mathcal D$ contains the eight benchmarks. Each benchmark’s weight equals its share of the evaluated responses, so a benchmark with more evaluated responses contributes more to Average. The aggregate answers how often a response is correct across this evaluation budget; the individual columns show performance on each task. For the three-seed 1.7B rows, this response-count-weighted accuracy is computed within each seed and then averaged. Base rows evaluate the corresponding base models.

Scores in the held-out evaluation table are percentages. The 1.7B rows report means and sample standard deviations over three seeds at 2,000 updates; standard deviation describes the spread across runs, and Average reports only the seed mean. The 4B checkpoints are evaluated at approximately 1,500 updates. Checkpoints are fixed independently of peaks in online evaluation. The table reports raw scores, while the learning curves use EMA $\alpha=0.20$ to trace training dynamics.

Independent held-out evaluation across eight benchmarks. Filter by model or compare methods within a matched setting; individual benchmark scores remain visible alongside Average, which does not display variance.

Across all seven matched off-policy comparisons, Proximal KL has higher Average than GRPO, with a largest absolute gain of 2.58 percentage points. The comparisons cover two model scales, two learning rates, and both fresh rollouts reused for multiple updates and stale rollout data. The aggregate improvement is measured on fixed checkpoints across the eight-benchmark evaluation suite.

AIME25 mean@16 favors GRPO in several comparisons, showing that the aggregate gain coexists with task-specific trade-offs. Each table column bolds the higher mean within a matched pair; both entries are bold when their unrounded gap is below 0.50 percentage points. This rule also applies to Average.

Fresh sampling and data reuse give different comparisons. With one update per fresh batch, the methods have comparable Average: 34.50 for GRPO and 34.09 for Proximal KL. GRPO has the higher aggregate in this on-policy comparison, while Proximal KL improves OlympiadBench, MinervaMath, and AIME24 mean@16. With fresh rollouts reused across four minibatches, Proximal KL’s Average is higher, 33.13 versus 31.99. Reusing those responses develops learner–data mismatch after the first update.

Accuracy at higher staleness. For 4B, we report the staleness-16 score minus the staleness-4 score across the same eight benchmarks. Average changes by +0.22 percentage points for Proximal KL and −1.04 for GRPO, computed from unrounded scores. Proximal KL’s Average rises from 48.13 to 48.35, while GRPO’s falls from 47.17 to 46.13. AIME24 mean@16 changes by +2.71 and −0.42 points, respectively. Proximal KL has the more favorable signed change on six of the eight benchmarks.

These are comparisons of complete training configurations. At 4B, Proximal KL uses a 32-token unit-weight prefix with a linearly decayed tail at staleness 4, and $H=8$ at staleness 16; both use $\beta=0.25$ and $\epsilon_F=0.05$. The higher-staleness configuration retains the accuracy advantage over GRPO. The mechanism ablations below examine individual update choices.

These comparisons test the practical update as a whole. Next we separate its ingredients: current-token correction, nonlinear clipping, cost bounds, and the reach of future credit.

Mechanism ablations

Starting from the full-tail working default, we examine which update choices sustain its gains and which simple adjustments can improve it. The ablations test current-token importance correction, GRPO clipping width and baseline sensitivity, the bound on individual future costs, and the correction horizon. Bounded costs work across a useful range of thresholds. Intermediate horizons can improve accuracy, and a separate fixed-policy diagnostic measures how shortening the sum reduces scalar correction variance.

Importance correction

The first ablation asks how sampling correction affects an update with the same Future-KL credit. To isolate that choice, this comparison fixes the model at 1.7B, the learning rate at $5\times10^{-6}$, staleness at 16, and the Future-KL configuration at full tail with $\beta=0.30$. All three updates retain the same form of token-specific future credit. Only the sampling weight changes.

Token-wise IS uses the current action’s weight $w_{m,i}$. Sequence-level IS uses the geometric mean of the probability ratio for the whole response. That weight is detached and multiplies each token’s own coefficient:

\[\bar W_m =\left( \frac{\pi_\theta(a_{m,1:T_m}\mid x)} {\pi_{\mathrm{behav},m}(a_{m,1:T_m}\mid x)} \right)^{\class{pkl-math-focus}{1/T_m}}.\]

The IS-free control sets the weight to 1. Both controls retain each token’s own $\widehat A_m-C_{m,i}^{(H)}$ and use their stated weight as the sole importance factor. The comparison changes how sampling correction enters the update while preserving each token’s future credit.

Exact response-level change of measure uses the full probability-ratio product: multiplying a response’s behavior probability by that ratio recovers its learner probability. Taking its $T_m$-th root generally breaks this equality, so averaging the reweighted responses can give a different expectation. The exponential-variance calculation in the off-policy theory section describes the full product. Here we directly measure training with the geometric-mean and IS-free weights.

All three updates retain token-specific Future-KL credit. Only their sampling weights change.

Token-wise correction sustains stronger training pass@16 and achieves higher MATH500 greedy pass@1 and AIME24 mean@16 than both controls. The controls lose substantial held-out accuracy while retaining future credit. These results support the practical role of current-action sampling correction alongside future feedback.

GRPO clipping width

GRPO’s clipping interval controls how far the policy ratio can move before its contribution is clipped. We compare $\epsilon_{\mathrm{clip}}=0.2$ and $0.4$, which give intervals $[0.8,1.2]$ and $[0.6,1.4]$. Both runs use Qwen3-1.7B, learning rate $10^{-5}$, controlled staleness 4, four optimizer minibatches, and the same reward normalization. The three panels track training mean@16, training pass@16, and raw gradient norm before optimizer clipping.

GRPO clipping widths 0.2 and 0.4. Both runs completed 1,000 updates. Accuracy uses EMA 0.20; gradient norms are raw and plotted on a logarithmic axis.

The wider interval produces larger gradient excursions together with a temporary decline in training mean@16 and pass@16. Width 0.2 maintains a steadier rise in training mean@16 with smaller gradient excursions. This comparison supports retaining 0.2 for the reported GRPO baseline. Appendix C.5 of the paper gives the clipping-width experiment.

Baseline perturbation

Sampling correction determines how an answer is weighted. The update also depends on where its advantage sits relative to zero. Group-relative advantages are computed from the sampled rewards, and their signs determine which contributions PPO clipping removes. Shifting every advantage in a group upward can turn negative advantages positive, changing which token contributions the clipping gate switches off.

We add 1 to each normalized advantage in groups with nonconstant rewards:

\[\widehat A'_{g,m}=\widehat A_{g,m}\class{pkl-math-focus}{+I_g}, \qquad I_g=\mathbf 1\{\max_m R_{g,m}>\min_m R_{g,m}\}.\]

This leaves verifier rewards and within-group advantage differences unchanged, modeling a common baseline shift. Proximal KL computes future costs as usual and subtracts them after the shift. Comparing each method with its own unperturbed run isolates its observed sensitivity relative to its original performance.

Solid and dashed lines show clean and perturbed advantages. Shading represents the within-method score difference.

Across the three displayed MATH500 and AIME24 metrics, Proximal KL is less sensitive to the intervention. This supports a practical advantage of additive future feedback over a gate that depends on the advantage sign. The shading shows the observed score difference between each method’s clean and perturbed runs.

To understand how the advantage sign can affect the update’s mean, consider a local calculation. Fix the proximal ratio above its upper bound, for example $\rho>1.2$. Suppose the true advantage is zero and its estimation error takes values $+\sigma$ and $-\sigma$ with equal probability. Without clipping, the coefficient has mean zero and variance $\sigma^2$.

PPO removes the positive contribution at this ratio and retains the negative one. The coefficient becomes equally likely to be $0$ or $-\sigma$:

\[\mathbb E[q_{\mathrm{clip}}]=\class{pkl-math-focus}{-\frac{\sigma}{2}}, \qquad \operatorname{Var}(q_{\mathrm{clip}})=\class{pkl-math-focus}{\frac{\sigma^2}{4}}.\]

Clipping shifts the conditional mean away from the true zero advantage, introducing bias in this coefficient even as its conditional variance decreases. The factor-of-four reduction applies to this local coefficient calculation, where the common importance-weighted score has been factored out. Variance of the full optimization update also depends on the score and its correlations with the coefficient.

We can also track an added baseline shift directly through the ideal unclipped update. With token-wise weighting, a history-only shift cancels in expectation. For a detached shift $b(h)$ that is fixed across current actions at a given history,

\[\mathbb E_{a\sim\pi_{\mathrm{behav}}} \left[\frac{\pi_\theta(a\mid h)}{\pi_{\mathrm{behav}}(a\mid h)} b(h)\nabla\log\pi_\theta(a\mid h)\right] =b(h)\nabla\sum_a\pi_\theta(a\mid h)=0.\]

An additive KL cost unchanged by the shift preserves this cancellation. The experimental $I_g$ depends on sampled group rewards: changing a generated answer can change whether its group has unequal rewards and therefore whether the shift is applied. The history-only identity holds a shift fixed across current actions; the experiment measures a group-dependent intervention. Its effect is the change between the clean and perturbed trajectories shown above.

Future-KL cost bounds

The working default bounds each future log-ratio cost at $\epsilon_F=0.05$ before adding it to the correction. A cost inside the interval is retained; a cost outside it is replaced by the nearest bound. We examine this choice using 1.7B at learning rate $5\times10^{-6}$ and staleness 4, fixing full tail and $\beta=0.30$ while varying only $\epsilon_F$. The panels show AIME24 mean@16, pass@16, and gradient norm before optimizer clipping. This norm measures the size of the computed loss gradient; optimizer clipping subsequently rescales the whole gradient if it exceeds its threshold. The future-cost bound acts earlier, on the terms used to compute that gradient.

Only the per-token cost bound changes. Gradient norms are measured before optimizer clipping; the unbounded configuration still retains Future-KL regularization.

The default $\epsilon_F=0.05$ achieves good AIME24 mean@16 and pass@16, with smaller pre-clipping gradient excursions than $2.0$ or no bound. The moderately larger bound $0.5$ also retains good accuracy, showing that useful regularization spans more than one precise threshold. The unbounded version retains Future-KL regularization and removes only the bound on each cost.

These results support bounding individual future costs before they accumulate in the correction. Within the tested range, increasing the bound further does not consistently improve accuracy.

Plotting norms before optimizer clipping is important here. Otherwise, spikes of very different magnitudes would sit against the same ceiling, obscuring whether $\epsilon_F$ changed the stochastic signal reaching the optimizer.

Future-KL horizon

Starting from the full-tail working default, we ask whether a shorter horizon can improve accuracy. At 1.7B, learning rate $5\times10^{-6}$, and staleness 4, we compare full tail, intermediate windows, and a one-token correction. The comparison includes fixed-$\beta$ settings and a pair with matched $\beta H$: the first tests horizon at the default coefficient, and the second tests whether increasing the local coefficient compensates for removing future terms. Truncation omits part of the ideal future credit and can also change correction fluctuations, so we examine accuracy and fixed-policy variance separately.

Learning curves compare the reach and strength of future credit. The fixed-policy diagnostic uses the same responses at every horizon and measures scalar correction variance.

Both $H=64$ configurations outperform full tail on late AIME24 mean@16 and pass@16 over their common observation interval, including a comparison keeping $\beta=0.30$ unchanged. This fixed-$\beta$ comparison provides evidence of a horizon benefit within the tested setting. Meanwhile, $(H,\beta)=(1,4.80)$ and $(64,0.075)$ both have $\beta H=4.8$, and the former improves mean@16 more slowly. Credit from later positions contributes useful information beyond the strength of the current-token penalty.

The intermediate window improves on the full-tail default in these training comparisons. The fixed-policy diagnostic below measures correction fluctuations independently: it holds policies and responses fixed while changing only horizon. These measurements establish the variance effect on that data; they do not identify it as the cause of the training-accuracy gain.

We measure the variance side separately. Training changes the policy as well as the observed responses. To isolate the effect of horizon on correction fluctuations, fix a separate retained pair of Qwen3-1.7B checkpoints and a shared bank of tokenized responses: 16 prompts with 16 responses per prompt. Fix $\beta=0.30$ and $\epsilon_F=0.05$. For each response, measure how its first-token scalar correction changes with $H$. The first subscript identifies the prompt and the second its sampled answer. We sum the clipped log-ratio costs from the first token up to the chosen horizon or the end of that answer, whichever comes first:

\[C_{pm}^{(H)} =\beta\sum_{j=1}^{\min(H,T_{pm})} \operatorname{clip}(\log\rho_{pm,j},-\epsilon_F,\epsilon_F).\]

Different prompts can have different mean corrections. To measure variation among answers to the same problem, first subtract each prompt’s mean correction from its individual response corrections. Sum the squared deviations and divide by the number of responses minus one to obtain that prompt’s sample variance. Finally, average these variances equally across prompts. In the formula, the uppercase counts refer to prompts and responses per prompt:

\[\overline C_p^{(H)}=\frac1M\sum_m C_{pm}^{(H)}, \qquad \widehat V(H)=\frac1P\sum_p\frac1{M-1}\sum_m \bigl(C_{pm}^{(H)}-\overline C_p^{(H)}\bigr)^2.\]

Only the extent of the sum changes. Policies remain fixed, every horizon uses the same responses, and variance is computed within each prompt before averaging across prompts. The interactive plot shows raw measured $\widehat V(H)$ without rescaling or smoothing. Over the tested horizons it increases with $H$, so truncating the tail reduces scalar correction fluctuations in this diagnostic.

The measured quantity is scalar correction variance: how much a single numerical correction changes between sampled responses. In a policy gradient, that correction multiplies an importance-weighted score vector whose magnitude and direction also vary across tokens and responses. Large corrections may coincide with large score vectors, or with vectors that cancel when summed. The full update also contains reward advantages and sums contributions across positions. Its parameter-gradient variability depends on these joint fluctuations.

Why call it a bias–variance trade-off? The derivation supplies the bias argument: finite $H$ omits a definite part of the full future sum. The fixed-policy diagnostic supplies variance evidence: shorter sums fluctuate less on the same data. Training curves show how the choice affects accuracy. These three pieces of evidence play different roles in the mechanism explanation.

To identify the bias source separately from cost clipping, consider the unbounded costs. The omitted tail is

\[C_i^{(\mathrm{full})}-C_i^{(H)} =\class{pkl-math-focus}{\beta\sum_{j=i+H}^{T}\ell_j}.\]

At position $i$, removing this cost changes the expected token-wise update by $\mathbb E_{\mathrm{behav}}[w_i\psi_i(C_i^{(\mathrm{full})}-C_i^{(H)})]$. A small value of this vector means a small change at that position; a large value means a substantial truncation effect. The relevant quantity includes the importance weight, the score vector, and the omitted tail together. The task and policies determine their joint contribution.

Variance of a sum includes both individual variances and pairwise covariances. Costs that rise and fall together across responses reinforce fluctuations; costs moving in opposite directions can offset them. The direction of the measured change therefore depends on these joint fluctuations. The increasing curve here describes the tested policies, responses, and horizons.

On fixed data, scaling $\beta$ multiplies every correction and its deviation from the mean by the same factor. Variance averages squared deviations, giving the $\beta^2$ scaling. Changing $\beta$ during training also changes the policy trajectory, so learning-curve differences cannot be extrapolated from this identity alone. The threshold $\epsilon_F$ changes costs nonlinearly: moving a bound changes the previously clipped terms while leaving costs inside the interval unchanged.

Matching $\beta H$ controls a rough maximum accumulation scale. With a fixed per-token bound, the sum’s absolute value is at most that bound times the number of included terms, multiplied by the penalty coefficient. Matching the coefficient–horizon product therefore matches this upper bound when enough tokens remain. Actual corrections also depend on which positions contribute and whether their signed costs reinforce or cancel. A large coefficient at $H=1$ uses the current token alone; a smaller coefficient at $H=64$ includes later positions. The two choices therefore retain distinct information even with equal upper bounds.

Why KL can help

KL 如何帮助优化

An on-policy example

一个 on-policy 例子

Proximal objective

Proximal objective

Future-KL credit assignment

Future-KL credit assignment

Stationary target

Stationary target

Token-wise importance correction

Token-wise importance correction

Pathwise cancellation

Pathwise cancellation

Uniqueness by backward induction

从后向前证明唯一性

Connection to Q-learning

与 Q-learning 的联系

Convergence

收敛

Stationary points and update dynamics

Stationary point 与更新过程

Convergence to the target

向 target 收敛

Improvement across reference refreshes

刷新 reference 后的改善

Finite-sample reward guarantee

有限样本奖励保证

Practical updates

实际更新

Proximal KL algorithm

Proximal KL 算法

Horizon and cost bounds

Horizon 与 cost 上限

Different Algorithms

不同算法

Reference refresh

Reference 刷新

Main results

Main results

Stability tests

稳定性测试

Held-out evaluation

Held-out evaluation

Mechanism ablations

机制消融

Importance correction

Importance correction

GRPO clipping width

GRPO clipping width

Baseline perturbation

Baseline 扰动

Future-KL cost bounds

Future-KL cost 上限

Future-KL horizon

Future-KL horizon