Reinforcement Learning in LLMs

Reinforcement learning has become an important part of LLM post-training. SFT teaches a model to imitate existing high-quality responses, while RL lets the model generate its own responses and continue learning from the feedback they receive. Intuitively, we want the model to gradually increase the probability of behaviors that obtain higher rewards.

That goal sounds straightforward, but turning it into a parameter update introduces problems that do not appear in SFT. Let us first recall a familiar SFT training step. Given a prompt $x$ and a demonstration response $y^{\star}=(y_1^{\star},\ldots,y_T^{\star})$, a causal language model predicts the next token at every position: the logits at the last prompt position predict $y_1^{\star}$, the logits after reading $y_1^{\star}$ predict $y_2^{\star}$, and so on. This one-position shift aligns each row of logits with its next-token target, while the loss mask removes the prompt and leaves only the cross-entropy over assistant response tokens:

\[\mathcal L_{\mathrm{SFT}}(\theta) =- \frac{1}{T} \sum_{t=1}^{T} \log p_\theta(y_t^{\star}\mid x,y_{<t}^{\star}).\]

Because the dataset fixes the target token at every position, SFT only needs the log probabilities of those tokens under their corresponding prefixes. The cross-entropy gradient can then travel through the logits and hidden states back to the model parameters.

In RL, the same decoder still produces a next-token distribution at every prefix, but RL gives these objects new names according to their roles in generation. Before generating token $t$, the prompt and the tokens generated so far form the current state $s_t=(x,y_{<t})$. The next token selected from the vocabulary is the action $a_t=y_t$. The probability distribution that the model assigns to all actions in this state is called the policy:

\[\pi_\theta(a_t\mid s_t) =p_\theta(y_t\mid x,y_{<t}).\]

The policy can therefore be viewed as a mapping from the current state to an action distribution. During an LLM rollout, we sample one token from this distribution, append it to the prefix to form a new state, and repeat the process until a complete response $y$ is produced:

\[y\sim\pi_\theta(\cdot\mid x).\]

A reward model, verifier, or environment then returns a scalar $R(x,y)$ for the response. Once the rollout is complete, the sampled response $y$ is fixed, so this reward is usually treated as fixed during the policy update. If we directly use $\mathcal L(\theta)=-R(x,y)$, its gradient with respect to the current policy parameters is zero:

\[\nabla_\theta[-R(x,y)]=0.\]

This seems to leave us with a contradiction: reward is the only feedback we receive, yet it cannot provide a gradient that can be backpropagated. RL can still use this scalar to update the model. The key distinction is that although the reward is fixed, the probability of generating the response is still determined by the model parameters. We will start from this distinction and derive the basic policy gradient used in LLM reinforcement learning.

From Expected Reward to Policy Gradient

Start with a tiny model that has only two possible responses. For the same prompt, let response $y^{(1)}$ have reward $1$ and response $y^{(2)}$ have reward $0$. Suppose the current policy generates $y^{(1)}$ with probability $p_\theta$. The probability of generating $y^{(2)}$ is then $1-p_\theta$, so the expected reward is

\[J(\theta) =p_\theta\cdot 1+(1-p_\theta)\cdot 0 =p_\theta.\]

The rewards of the two responses are fixed constants, yet the expected reward changes with $p_\theta$. For example, if the policy raises the probability of the high-reward response from $0.25$ to $0.30$, its expected reward also rises from $0.25$ to $0.30$. RL optimizes precisely this average performance determined by the whole policy distribution.

Now generalize the example to any number of responses. For a given prompt $x$, the policy objective can be written as

\[J(\theta) =\mathbb E_{y\sim\pi_\theta(\cdot\mid x)}[R(x,y)] =\sum_y\pi_\theta(y\mid x)R(x,y).\]

This expectation defines the quantity we want to optimize: the average reward of the policy over all possible responses. The second equality expands it into an explicit sum, where $y$ ranges over every complete response that the policy could generate for the prompt. Each $R(x,y)$ can remain fixed, while $J(\theta)$ still depends on the model parameters through $\pi_\theta(y\mid x)$. Changing the probabilities of different responses redistributes the probability mass in the sum and changes the expected reward.

Once the objective is defined, training needs to know how to change $\theta$, so the quantity we really need is $\nabla_\theta J(\theta)$. There is also a practical limitation: an LLM’s response space cannot be enumerated, so training can only roll out a small number of responses from the current policy. If we can write the gradient itself as an expectation under the current policy, we can approximate it with an average over these samples. This goal determines the direction of the derivation.

Start from the explicit sum for the expected reward. Since the reward associated with each response is treated as a constant, the gradient acts only on the probability with which the policy selects it:

\[\nabla_\theta J(\theta) =\sum_y R(x,y)\nabla_\theta\pi_\theta(y\mid x).\]

This is a correct gradient, but it cannot yet be estimated directly from samples. By the definition of expectation, expanding a quantity estimated by sampling from $\pi_\theta$ produces terms weighted by $\pi_\theta(y\mid x)$. The current gradient sum instead contains $\nabla_\theta\pi_\theta(y\mid x)$. We therefore need to rewrite it as $\pi_\theta(y\mid x)$ multiplied by another computable quantity. Log probability gives exactly this factorization. From the derivative of $\log z$,

\[\nabla_\theta\log\pi_\theta(y\mid x) =\frac{1}{\pi_\theta(y\mid x)} \nabla_\theta\pi_\theta(y\mid x).\]

Multiplying both sides by $\pi_\theta(y\mid x)$ gives

\[\nabla_\theta\pi_\theta(y\mid x) =\pi_\theta(y\mid x) \nabla_\theta\log\pi_\theta(y\mid x).\]

Substituting this identity into the gradient of the objective yields

\[\nabla_\theta J(\theta) =\sum_y \pi_\theta(y\mid x) \left[ R(x,y)\nabla_\theta\log\pi_\theta(y\mid x) \right].\]

Now every term contains the desired $\pi_\theta(y\mid x)$ factor. By the definition of expectation, the whole gradient can be written as

\[\nabla_\theta J(\theta) =\mathbb E_{y\sim\pi_\theta(\cdot\mid x)} \left[ R(x,y)\nabla_\theta\log\pi_\theta(y\mid x) \right].\]

This is the form we wanted at the start: the gradient itself is an expectation under the policy. We can roll out a response $y$ from the current policy, obtain its reward $R(x,y)$, and compute the log probability of that response under the model to construct a stochastic gradient estimate:

\[\widehat{\nabla_\theta J} =R(x,y)\nabla_\theta\log\pi_\theta(y\mid x), \qquad y\sim\pi_\theta(\cdot\mid x).\]

Averaging this estimate over many sampled responses gradually approaches the true $\nabla_\theta J(\theta)$. This is the central result of policy gradient (Williams, 1992). The reward does not need to be differentiable with respect to $\theta$. It acts as a weight that determines how large an update the sampled response’s log probability should receive. Under gradient ascent, a positive reward increases the log probability of the response. Implementations usually use gradient descent, so the corresponding loss can be written as

\[\mathcal L_{\mathrm{PG}}(\theta) =-R(x,y)\log\pi_\theta(y\mid x),\]

where $R(x,y)$ is treated as a constant during backpropagation. The gradient travels through $\log\pi_\theta(y\mid x)$ back to the model parameters, reconnecting with the same token-log-probability path used by SFT.

For an autoregressive LLM, the probability of a complete response is the product of the next-token probabilities at each step:

\[\pi_\theta(y\mid x) =\prod_{t=1}^{T} \pi_\theta(y_t\mid x,y_{<t}).\]

Taking the logarithm turns the product into a sum over tokens:

\[\log\pi_\theta(y\mid x) =\sum_{t=1}^{T} \log\pi_\theta(y_t\mid x,y_{<t}).\]

The most basic sequence-level policy-gradient loss can therefore be written as

\[\mathcal L_{\mathrm{PG}}(\theta) =-R(x,y) \sum_{t=1}^{T} \log\pi_\theta(y_t\mid x,y_{<t}).\]

This form is already very close to the SFT loss. Both backpropagate through the log probabilities of sampled or target tokens. In policy gradient, the training direction and strength of each token are weighted by the signal obtained from the rollout. An otherwise non-differentiable sequence-level reward has now become a gradient that can be computed with respect to the model parameters.

From Sequence Reward to Token-Level Advantage

The policy-gradient loss from the previous section weights the sampled response’s log probability by its reward:

\[\mathcal L_{\mathrm{PG}}(\theta) =-R(x,y) \log\pi_\theta(y\mid x).\]

Start by treating the complete response as one sampled outcome. Suppose the current policy generates two responses for the same prompt and receives scores of $80$ and $90$. If it usually scores only $60$, both responses are above its current level. If it usually scores $85$, the $90$ response is still worth encouraging, while the $80$ response is below expectation. The absolute reward lacks this reference point. What the policy update really cares about is how much better a response is than the policy’s usual performance.

For a prompt $x$, choose a baseline $b(x)$ and define a response-level advantage as the reward minus the baseline:

\[A(x,y)=R(x,y)-b(x).\]

As long as $b(x)$ does not depend on the sampled response $y$ and is treated as a fixed coefficient during the policy update, subtracting it does not change the expected policy gradient. Using the identity $\pi_\theta\nabla_\theta\log\pi_\theta=\nabla_\theta\pi_\theta$ derived above, the expected contribution of the baseline is

\[\begin{aligned} \mathbb E_{y\sim\pi_\theta(\cdot\mid x)} \left[ b(x)\nabla_\theta\log\pi_\theta(y\mid x) \right] &=b(x)\nabla_\theta\sum_y\pi_\theta(y\mid x) \\ &=b(x)\nabla_\theta 1=0. \end{aligned}\]

The policy gradient can therefore use $A(x,y)$ in place of the raw reward:

\[\nabla_\theta J(\theta) =\mathbb E_{y\sim\pi_\theta(\cdot\mid x)} \left[ A(x,y)\nabla_\theta\log\pi_\theta(y\mid x) \right].\]

If $b(x)$ approximates the current policy’s expected reward for prompt $x$, a positive advantage means that the response exceeded the current level, while a negative advantage means that it fell below it. In the previous example, taking $b(x)=85$ gives advantages of $-5$ and $5$. The baseline does not change the expected policy gradient, but it makes each sample’s coefficient directly express relative performance and usually reduces the variance of the gradient estimate.

When a Response Is Split into Tokens

So far, we have treated a complete response as one choice. In an LLM, the policy is actually made up of token-by-token distributions:

\[\log\pi_\theta(y\mid x) =\sum_{t=1}^{T} \log\pi_\theta(y_t\mid x,y_{<t}).\]

If we use the same response-level advantage directly, every sampled token still shares the same coefficient. Yet each generated token gives the model more information about the response. A baseline that sees only the prompt cannot reflect this new information. What we really want to know is: after generating the current token, how much better or worse is the response’s prospect of receiving a high reward?

Consider a common setting in LLM RL: the model generates a response token by token, no intermediate rewards are provided during generation, and a verifier returns the final reward $R(x,y)$ only after the response ends. At timestep $t$, the model has seen the prompt and the first $t-1$ response tokens. Denote this prefix by

\[s_t=(x,y_{<t}).\]

Starting from the same prefix, the policy can generate many different continuations, and those continuations may receive different final rewards. Averaging the rewards over these possible continuations gives the value of the current prefix:

\[V^\pi(s_t) =\mathbb E_\pi[R(x,y)\mid x,y_{<t}],\]

where the continuation in the expectation is generated by the current policy. $V^\pi(s_t)$ represents the final reward the policy expects to obtain if it continues from the current prefix. The response-level baseline $b(x)$ uses only the prompt. $V^\pi(s_t)$ also uses the tokens already generated, so it can update as the response progresses.

The interpretation is especially direct when the verifier reward is binary. The average reward is then the success probability, so $V^\pi(s_t)=0.6$ means that continuing from the current prefix has a $60\%$ probability of eventual success.

Before sampling the next token, the response’s prospects are represented by $V^\pi(s_t)$. The policy then samples a token $a_t=y_t$ from $\pi(\cdot\mid s_t)$ and appends it to the response. The new prefix is

\[s_{t+1}=(x,y_{\leq t}).\]

The response’s prospects are now represented by $V^\pi(s_{t+1})$. Suppose the current prefix has value $0.6$, and after sampling a particular token the value of the new prefix becomes $0.75$. In the binary-reward example, the model now believes that the probability of eventual success has risen from $60\%$ to $75\%$. Relative to the baseline before sampling, this token brought a gain of $0.15$, so its advantage is $0.15$. If the new prefix’s value falls to $0.4$, its advantage is $-0.2$.

This gives us the central token-level intuition in a terminal-reward setting. For a non-terminal token, the difference in value before and after generation is its exact advantage:

\[A^\pi(s_t,a_t) =V^\pi(s_{t+1})-V^\pi(s_t).\]

Standard RL notation calls the expected reward after fixing the current action the action value:

\[Q^\pi(s_t,a_t) =\mathbb E_\pi[R(x,y)\mid x,y_{\leq t}].\]

In an LLM, choosing token $a_t$ determines the next prefix $s_{t+1}$. For a non-terminal token, generation has not returned a reward yet, so

\[Q^\pi(s_t,a_t)=V^\pi(s_{t+1}).\]

The $Q-V$ advantage commonly used in RL is therefore exactly the prefix-value difference we just described:

\[A^\pi(s_t,a_t) =Q^\pi(s_t,a_t)-V^\pi(s_t).\]

For the final token, the response ends immediately and produces the final reward, so $Q^\pi(s_T,a_T)=R(x,y)$ and the corresponding advantage is $R(x,y)-V^\pi(s_T)$. $Q$ gives us a single formula that covers ordinary and terminal tokens. The baseline we need to learn remains the prefix value.

Using $V^\pi(s_t)$ as a baseline does not change the expected policy gradient because it is determined by the prefix before the current token is sampled and does not depend on the sampled action. The token-level policy gradient can be written as

\[\nabla_\theta J(\theta) =\mathbb E_{y\sim\pi_\theta(\cdot\mid x)} \left[ \sum_{t=1}^{T} A^\pi(s_t,a_t) \nabla_\theta\log\pi_\theta(a_t\mid s_t) \right].\]

We now need to distinguish the exact advantage from the estimate available during training. Exact $Q^\pi(s_t,a_t)$ requires the expected reward over every possible continuation after the fixed current token. The continuation space of a real LLM cannot be enumerated. In practice, we train a parameterized value model $V_\phi(s_t)$ to approximate $V^\pi(s_t)$. Given a fixed prompt-response sequence, a causal Transformer produces a hidden state $h_t$ for every prefix $s_t$, and a scalar value head outputs

\[V_\phi(s_t)=w_V^\top h_t+b_V.\]

The causal mask ensures that $h_t$ can read only the current prefix, so a single forward pass over the full sequence can produce values for every response position. The value model may use a separate Transformer or share part of the backbone with the policy. Regardless of the implementation, it outputs a scalar expected return for each prefix. $Q^\pi(s_t,a_t)$ is estimated from a finite number of rollouts.

Suppose we continue from the same $s_t,a_t$ many times and average the final rewards of the resulting continuations. This average approaches $Q^\pi(s_t,a_t)$. A single rollout provides only one random outcome, so its final $R(x,y)$ can be viewed as one Monte Carlo sample of $Q^\pi(s_t,a_t)$. Using this sample to estimate $Q$ and subtracting the learned baseline gives the Monte Carlo advantage:

\[\hat A_t^{\mathrm{MC}} =R(x,y)-V_\phi(s_t).\]

All positions in the same response use the same $R(x,y)$, while the prefix-dependent baseline $V_\phi(s_t)$ changes along the response, so different tokens can still receive different estimated advantages. At the same time, an early token may happen to be followed by an excellent continuation or be dragged down by poor later choices. The single-rollout estimate assigns all of this later sampling randomness to the current token, so its variance is high, and it is difficult to distinguish the effect of the current token from the effect of the continuation that follows it.

How GAE Combines Estimates at Different Horizons

The quantity $R(x,y)-V_\phi(s_t)$ from the previous section is already one estimate of the current token’s advantage: $R(x,y)$ is the final return obtained after choosing this token, and $V_\phi(s_t)$ is the expected return before generating it. The problem is that $R(x,y)$ comes from one particular sampled continuation. Every sampling decision after the current token affects the final result, so using this one $R(x,y)$ to estimate how good the current token was often has high variance.

The value model provides another way to estimate the return. We can use the reward observed at the current step and let $V_\phi(s_{t+1})$ predict the remaining return. We can also use the next two observed rewards and let $V_\phi(s_{t+2})$ predict the rest. Continuing to increase the number of observed rewards gives one-step, two-step, and eventually full-rollout estimates that use the complete response. Shorter estimates rely on value prediction earlier, while longer estimates include more of the sampled continuation. GAE combines these advantage estimates from different horizons (Schulman et al., 2015).

First define the object we want to estimate. Let the reward obtained immediately after taking action $a_t$ be $r_t$. The discounted return starting at timestep $t$ is

\[G_t =r_t+\gamma r_{t+1}+\gamma^2r_{t+2}+\cdots+ \gamma^{T-t}r_T.\]

$\gamma$ determines how much weight delayed rewards receive in the return. A reward of $1$ that appears two steps later contributes $\gamma^2$ to the current return. In LLMs, the outcome reward usually appears only at the end of the response, so $r_1,\ldots,r_{T-1}$ are $0$ and $r_T=R(x,y)$. Training commonly uses $\gamma=1$, so the final reward does not decay with token distance.

Taking the current reward out of the return definition gives the most important step in the derivation:

\[G_t=r_t+\gamma G_{t+1}.\]

After the current action is executed, $r_t$ is observable, while $G_{t+1}$ still depends on the continuation that has not yet been generated. The next-state value $V^\pi(s_{t+1})$ predicts exactly this future return:

\[V^\pi(s_{t+1}) =\mathbb E_\pi[G_{t+1}\mid s_{t+1}].\]

We can therefore estimate the current action’s return with the current reward plus a predicted future return. Subtracting the baseline before the action gives the one-step advantage estimate, also called the TD error:

\[\delta_t =r_t+\gamma V_\phi(s_{t+1})-V_\phi(s_t).\]

Here $r_t$ is the reward already obtained by the current transition. $V_\phi(s_{t+1})$ predicts the entire future return beginning at the next step, so from timestep $t$ onward this future tail must be multiplied by $\gamma$. That is why $\gamma$ appears in front of the value term.

The one-step estimate begins relying on value prediction at $s_{t+1}$. We can observe one more step, use the actual $r_{t+1}$, and bootstrap only at $s_{t+2}$:

\[\begin{aligned} \hat A_t^{(2)} &=r_t+\gamma r_{t+1} +\gamma^2V_\phi(s_{t+2})-V_\phi(s_t) \\ &=\delta_t+\gamma\delta_{t+1}. \end{aligned}\]

Extending the horizon by one step adds one more observed reward and pushes the value prediction one step farther into the future. If the environment returns $1$ at every step and $\gamma=1$, the observed-reward part of the two-step estimate is indeed $1+1=2$. The complete advantage also adds $V_\phi(s_{t+2})$ and subtracts the original $V_\phi(s_t)$. In a terminal-reward LLM, the intermediate rewards are $0$, and $R(x,y)$ enters the sum only when the horizon reaches the end of the response.

Short-horizon estimates rely more heavily on the value model and usually have lower variance. Long-horizon estimates use more of the sampled continuation and gradually approach the full-rollout estimate $G_t-V_\phi(s_t)$, while bringing in more sampling variance. GAE uses $\lambda$ to form an exponentially weighted combination of these different horizons. Every additional TD error kept in the estimate contributes one more factor of $\lambda$. In terms of TD errors,

\[\hat A_t^{\mathrm{GAE}(\gamma,\lambda)} =\delta_t +\gamma\lambda\delta_{t+1} +(\gamma\lambda)^2\delta_{t+2} +\cdots.\]

$\gamma$ and $\lambda$ play different roles here. $\gamma$ already determines the value of future rewards in the definition of return. $\lambda$ determines how much long-horizon information the estimator keeps, trading off value-model bias against rollout variance. For the TD error $k$ steps in the future, $\gamma^k$ comes from its position in the return, while $\lambda^k$ comes from the horizon weighting. In the common LLM setting with $\gamma=1$, the coefficient becomes $\lambda^k$, making the distinction especially direct.

When $\lambda=0$, GAE uses only the one-step TD error. When $\lambda=1$, it keeps all subsequent TD errors. If the future value of the terminal state is set to zero, the intermediate value terms cancel one another, leaving the full-rollout estimate $G_t-V_\phi(s_t)$. The policy-gradient loss then becomes

\[\mathcal L_{\mathrm{PG}}(\theta) =- \sum_{t=1}^{T} \hat A_t \log\pi_\theta(a_t\mid s_t).\]

Return to the original question of this section: we want to estimate how much the current token changes the response’s expected return. The full-rollout estimate uses the actual rollout all the way to the response ending. The one-step estimate uses only the reward immediately in front of us and then lets the value model predict the remaining return. The two-step estimate observes one additional reward before handing the prediction to the value model. GAE weights this whole family of estimates, with $\lambda$ deciding whether the weighting favors shorter horizons or estimates that look farther along the sampled rollout.

GAE ultimately produces two closely related training quantities for the same prefix. $\hat A_t$ is the advantage relative to the rollout-time baseline and is used to train the policy. Adding the baseline back gives the regression target for the value model:

\[\hat R_t = \hat A_t+V_{\mathrm{old}}(s_t).\]

Here $V_{\mathrm{old}}(s_t)$ is the value prediction saved when GAE was computed. When $\lambda<1$, $\hat R_t$ is a target formed by mixing sampled rewards with value bootstrapping. The value model therefore provides prefix-dependent predictions inside GAE and learns a more accurate expected return from $\hat R_t$. The policy uses $\hat A_t$ as the update coefficient for each sampled token: a positive advantage raises the token’s probability, a negative advantage lowers it, and the magnitude determines the strength of the update signal.

Proximal Policy Optimization (PPO)

The policy-gradient loss from the previous section is

\[\mathcal L_{\mathrm{PG}}(\theta) = -\mathbb E_t\left[ \hat A_t\log\pi_\theta(a_t\mid s_t) \right].\]

If we generate rollouts with the current policy, compute the advantages, take one small gradient update, and immediately discard the data before rolling out again, this loss can be used directly. The samples and the gradient always come from the same policy. As long as the update is small, they describe an improvement direction near the policy’s current position.

In practice, training usually collects a batch of responses, computes rewards, old values, GAE advantages, and value targets, then splits the batch into minibatches and performs several optimizer steps, sometimes over multiple epochs. This makes better use of the rollouts that have already been generated. It also creates a problem: the rollout batch stays fixed throughout the update, while the policy being trained changes after every optimizer step.

Call the policy snapshot used to generate these responses $\pi_{\mathrm{old}}$, and call the policy currently being updated $\pi_\theta$. At the beginning of the update they are identical. After the first optimizer step, $\pi_\theta$ has changed, while the sampled tokens, visited prefixes, and advantages in the remaining minibatches still come from $\pi_{\mathrm{old}}$. Implementations usually save rollout-time log probabilities, so computing the ratio does not necessarily require keeping a second complete policy model in memory.

This gives us two connected questions. First, how should actions sampled from the old policy be used to estimate the objective under the current policy? Second, as the current policy moves away from the old policy, how much can we still trust this fixed evidence? PPO answers these questions with a probability ratio and clipping. “Proximal” means that this local evidence is used only near the rollout policy, after which a limited number of updates is followed by a new rollout (Schulman et al., 2017).

Probability Ratio: Reweighting Old-Policy Samples

Start with the first question. Fix a prefix $s$ in the rollout. The action in hand was sampled from the old policy, while the expectation we want to estimate is under the current policy. Suppose the old policy assigns an action probability of $0.2$ at this prefix, and the current policy assigns it $0.4$. If we repeatedly sample from both policies at the same prefix, the old policy produces this action about $20$ times per $100$ samples, while the current policy produces it about $40$ times. We still have only the roughly $20$ observations sampled by the old policy, yet we want to estimate the expectation under the current policy. Each observation therefore needs to represent about two occurrences in the current-policy world, which means multiplying it by the weight $\frac{0.4}{0.2}=2$.

This weight is the ratio between the current and old probabilities. We used $r_t$ for reward in the previous section, so we will denote the probability ratio by

\[\rho_t(\theta) = \frac{\pi_\theta(a_t\mid s_t)} {\pi_{\mathrm{old}}(a_t\mid s_t)}.\]

At this stage, $\rho_t$ serves as a change of measure: it lets old-policy samples estimate an expectation under another action distribution. It has not yet decided how far one update should move. To express this intuition formally, treat the old-policy advantage $A^{\pi_{\mathrm{old}}}(s,a)$ as a quantity that has already been computed:

\[\mathbb E_{a\sim\pi_\theta(\cdot\mid s)} \left[A^{\pi_{\mathrm{old}}}(s,a)\right] = \mathbb E_{a\sim\pi_{\mathrm{old}}(\cdot\mid s)} \left[ \frac{\pi_\theta(a\mid s)} {\pi_{\mathrm{old}}(a\mid s)} A^{\pi_{\mathrm{old}}}(s,a) \right].\]

The left side requires actions sampled from the current policy. The right side keeps using old-policy samples and adjusts each sample’s statistical weight with the ratio. Applying this identity to a rollout batch gives the local surrogate objective:

\[J_{\mathrm{sur}}(\theta) = \mathbb E_t\left[\rho_t(\theta)\hat A_t\right].\]

This objective also connects back to the policy gradient from the previous section. The old-policy probability is constant with respect to $\theta$, so

\[\nabla_\theta\rho_t = \rho_t\nabla_\theta\log\pi_\theta(a_t\mid s_t).\]

At the start of the update, $\pi_\theta=\pi_{\mathrm{old}}$, so $\rho_t=1$. Therefore

\[\nabla_\theta\left(\rho_t\hat A_t\right) = \hat A_t\nabla_\theta\log\pi_\theta(a_t\mid s_t),\]

which is exactly the sampled-token policy-gradient term from the previous section. Here $J_{\mathrm{sur}}$ is written as an objective to maximize. In practice, we minimize $-J_{\mathrm{sur}}$.

This correction has an important boundary. An LLM generates a response token by token, so changing an earlier token changes every later prefix. Suppose the old-policy rollout reached a prefix $s_t$. If the current policy lowers the probability of an earlier token, it may reach $s_t$ much less often, or never reach it at all. Yet when we compute $\rho_t$, we treat $s_t$ as given and compare only the two policies’ probabilities for the current action at that prefix.

The ratio therefore corrects the conditional action distribution: once we have arrived at this prefix, it tells us how to reweight the next token. It does not correct the probability of visiting the prefix itself. Which prefixes the rollout visited, which continuation appeared afterward, and the value of $\hat A_t$ are still determined by the old policy, so $J_{\mathrm{sur}}$ remains a local objective around the old policy. Intuitively, the ratio can recompute the next-step weight at a checkpoint on an old path, but it does not regenerate the path that led to the checkpoint.

Although the ratio is derived as an importance weight, its value also directly records how much the sampled action’s probability has changed. When $\rho_t=1$, the new and old probabilities are equal. $\rho_t=1.25$ means that the current policy has increased the action’s probability by $25\%$ relative to the old policy, while $\rho_t=0.8$ means a relative decrease of $20\%$. This interpretation does not change why the ratio multiplies the advantage statistically, but it lets us observe how far the current policy has moved along this old-policy evidence.

Clipping: Limiting the Influence of Fixed Rollout Evidence

The ratio solves how old-policy samples enter the current-policy expectation. It does not yet answer how far the same batch of fixed rollouts can keep pushing the current policy. To see the issue, temporarily add no extra restriction. For a positive-advantage sample, the corresponding contribution to the surrogate objective is $\rho_t\hat A_t$. Since we maximize this objective, raising the probability of the sampled action raises $\rho_t$ and makes this term larger. This is the surrogate term without clipping, often called the unclipped term.

Near the old policy, this is the direction we want. If an action has a positive estimated advantage, the policy should increase its probability. The problem appears when the same data is reused repeatedly. $\hat A_t$ remains the fixed estimate supplied by the old rollout, while the unclipped objective continues to reward increasing $\rho_t$ as the current policy moves farther away, as if this old evidence remained equally reliable at every distance. For a negative advantage, the objective similarly continues to encourage decreasing $\rho_t$.

The two roles of the ratio need to be kept separate. As a weight in an expectation, a large ratio is reasonable because the action really would occur more frequently under the current policy. As a variable in an optimization objective, a continually increasing ratio lets the same fixed evidence provide an increasingly strong incentive. The ratio corrects the sampling distribution, yet it does not give this evidence a useful range. As the current policy moves farther from the old policy, the match between the prefix distribution, later continuations, the advantage estimate, and the current policy becomes weaker.

This is the problem clipping addresses. The advantage stays fixed during the current iteration. PPO limits how much one piece of old-policy evidence can continue to contribute in its favorable direction. It constructs a copy of the ratio clipped to $[1-\epsilon,1+\epsilon]$ and chooses the more conservative of the original and clipped terms:

\[J_{\mathrm{PPO}}(\theta) = \mathbb E_t\left[ \min\left( \rho_t(\theta)\hat A_t, \operatorname{clip}\bigl(\rho_t(\theta),1-\epsilon,1+\epsilon\bigr)\hat A_t \right) \right].\]

The easiest place to get lost in this formula is understanding when $\min$ returns which candidate. Splitting it into cases can make the main line harder to see. Figure 1 puts the six possibilities for one sampled token together and shows why the clipped term becomes a plateau.

To keep the contribution of one sampled token separate from the full expectation, the vertical axis in the figure represents only the per-token term inside the brackets, denoted by $\ell_t$.

$\rho_t$ range	$\hat A_t$	Return value of $\min$	Objective clipped?	Sign of objective	policy gradient?
$1-\epsilon<\rho_t<1+\epsilon$	$+$	$\rho_t\hat A_t$	no	$+$	✓
$1-\epsilon<\rho_t<1+\epsilon$	$-$	$\rho_t\hat A_t$	no	$-$	✓
$\rho_t<1-\epsilon$	$+$	$\rho_t\hat A_t$	no	$+$	✓
$\rho_t<1-\epsilon$	$-$	$(1-\epsilon)\hat A_t$	yes	$-$	×
$\rho_t>1+\epsilon$	$+$	$(1+\epsilon)\hat A_t$	yes	$+$	×
$\rho_t>1+\epsilon$	$-$	$\rho_t\hat A_t$	no	$-$	✓

$\hat A_t>0$

$\hat A_t<0$

The figure shows that in the two clipped cases PPO returns the fixed terms $(1+\epsilon)\hat A_t$ or $(1-\epsilon)\hat A_t$. The term still has an objective value, yet it no longer depends on $\theta$, so its policy-gradient contribution is $0$.

The curves show the same behavior geometrically. When $\hat A_t>0$, the policy wants to raise the sampled action’s probability, and the selected objective becomes flat after $\rho_t=1+\epsilon$. When $\hat A_t<0$, the policy wants to lower the sampled action’s probability, and the plateau appears to the left of $\rho_t=1-\epsilon$.

Let $\epsilon=0.2$. If $\hat A_t=2$ and $\rho_t=1.3$, the original term is $2.6$ and the clipped term is $2.4$, so $\min$ returns $2.4$. The actor loss has value $-2.4$, yet increasing $\rho_t$ further does not change the returned value, so this sample’s policy gradient is $0$. If the policy moves in the direction opposed to the advantage, $\min$ returns the original term and the objective continues to penalize that change. Clipping removes only the extra gain available after moving too far in the favorable direction. It does not force every ratio to stay inside $[1-\epsilon,1+\epsilon]$.

KL and the Trust-Region View

Clipping builds a local safeguard from the ratio of one sampled action. If we want to describe policy change at the scale of the entire next-token distribution, we can take the logarithm of the ratio and write the relative probability change as a log-probability difference:

\[\log\rho(a\mid s) = \log\pi_\theta(a\mid s) - \log\pi_{\mathrm{old}}(a\mid s).\]

KL divergence aggregates log-probability differences over an entire action distribution. Its general definition is

\[D_{\mathrm{KL}}(p\Vert q) = \sum_a p(a)\log\frac{p(a)}{q(a)}.\]

The quantity gives more weight to actions that $p$ considers likely and compares their log probabilities under $p$ and $q$. For the new and old policies at the same prefix, it can be written as

\[D_{\mathrm{KL}} \left( \pi_{\mathrm{old}}(\cdot\mid s) \Vert \pi_\theta(\cdot\mid s) \right) = -\mathbb E_{a\sim\pi_{\mathrm{old}}(\cdot\mid s)} [\log\rho(a\mid s)].\]

The ratio describes the relative probability change of one sampled action, while KL describes the average change across the whole next-token distribution. KL is $0$ when the two distributions are identical. It is directional, so swapping the distributions usually gives a different value.

A trust region is a neighborhood around the old policy. Only within this neighborhood do we trust the surrogate built from old prefixes and old advantages to guide the current policy reliably. Trust Region Policy Optimization (TRPO) writes this idea directly as a constrained optimization (Schulman et al., 2015):

\[\begin{aligned} \max_\theta\quad &\mathbb E_t\left[\rho_t(\theta)\hat A_t\right] \\ \text{subject to}\quad &\mathbb E_{s_t}\left[ D_{\mathrm{KL}} \left( \pi_{\mathrm{old}}(\cdot\mid s_t) \Vert \pi_\theta(\cdot\mid s_t) \right) \right] \leq \delta. \end{aligned}\]

PPO keeps the same local-update intuition and uses clipping on sampled ratios to avoid this kind of constrained optimization. Clipping does not guarantee that all ratios fall inside the clipping interval, and it does not guarantee that the overall KL stays below $\delta$. It limits how much fixed rollout evidence can continue to contribute to the surrogate. KL gives the distribution-level trust-region view, while clipping provides a safeguard that is easier to optimize with minibatch SGD.

A Complete LLM PPO Update

At this point, PPO has three kinds of core objects. The current policy is updated, the old policy is the fixed snapshot used for this rollout, and the value model provides critic predictions. LLM RLHF usually adds a reference policy as well (Ouyang et al., 2022). These policies address different time scales. The old policy is the comparison point for the current PPO iteration, while the reference policy is usually the SFT model from before RL and constrains long-term drift across many rounds.

The models and parameter snapshots involved in one PPO update are:

Clipping constrains the local change between the current policy and the old policy for this iteration. At the end of each round, however, the updated policy becomes the next round’s old policy. After many small updates, the model may still drift far from the SFT model that initialized RL. The reference policy provides an anchor across many iterations. The reward model or verifier is also usually frozen and provides the task reward for each sampled response.

For now, use the convention that places the reference KL directly in the actor loss. The task reward enters GAE, while reference regularization stays in the actor update, so the two training signals remain separate.

The actor minimizes the clipped policy loss:

\[\mathcal L_{\mathrm{policy}}(\theta) = -\mathbb E_t\left[ \min\left( \rho_t(\theta)\hat A_t, \operatorname{clip}\bigl(\rho_t(\theta),1-\epsilon,1+\epsilon\bigr)\hat A_t \right) \right].\]

If the reference policy supplies the long-term regularization, we can add a reference penalty directly to the actor loss. For each rollout prefix, define

\[\mathcal R_{\mathrm{ref}}(\theta) = \mathbb E_t\left[ D_{\mathrm{KL}} \left( \pi_\theta(\cdot\mid s_t) \Vert \pi_{\mathrm{ref}}(\cdot\mid s_t) \right) \right].\]

This compares the complete next-token distributions of the current and reference policies at the same prefix. The farther the current policy moves from the reference, the larger $\mathcal R_{\mathrm{ref}}$ becomes. The actor loss is therefore

\[\mathcal L_{\mathrm{actor}}(\theta) = \mathcal L_{\mathrm{policy}}(\theta) + \beta\mathcal R_{\mathrm{ref}}(\theta).\]

The reference penalty acts only on the actor. It does not enter GAE or change the critic’s value target, which keeps PPO’s local clipping separate from the reference model’s long-term regularization.

The critic regresses toward the fixed value target constructed by GAE in the previous section:

\[\mathcal L_V(\phi) = \frac12\mathbb E_t\left[ \left(V_\phi(s_t)-\hat R_t\right)^2 \right].\]

Training may also include an optional entropy bonus to keep the next-token distribution from becoming too concentrated too early:

\[\mathcal H_t = -\sum_{v\in\mathcal V} \pi_\theta(v\mid s_t) \log\pi_\theta(v\mid s_t).\]

Under this convention, the overall training objective is

\[\mathcal L_{\mathrm{total}} = \mathcal L_{\mathrm{actor}} +c_V\mathcal L_V -c_H\mathbb E_t[\mathcal H_t].\]

Some LLM RLHF implementations use another convention and put the sampled reference KL into the rollout reward. For a sampled token $a_t$, they can use

\[r_t = r_t^{\mathrm{task}} - \beta \left[ \log\pi_{\mathrm{old}}(a_t\mid s_t) - \log\pi_{\mathrm{ref}}(a_t\mid s_t) \right].\]

In this case, the KL penalty enters GAE and therefore affects both the actor’s $\hat A_t$ and the critic’s $\hat R_t$. It still expresses reference regularization, but the regularization travels through the reward, advantage, and value target.

This choice is more than an engineering convenience. Putting the reference KL into the reward is equivalent to directly optimizing the KL-regularized RL objective

\[\max_{\pi}\; \mathbb E_{x\sim\mathcal D,\;y\sim\pi(\cdot\mid x)} \left[ r^{\mathrm{task}}(x,y) \right] - \beta\, \mathbb D_{\mathrm{KL}} \left[ \pi(\cdot\mid x) \,\|\, \pi_{\mathrm{ref}}(\cdot\mid x) \right],\]

which has a closed-form optimum with respect to $\pi$ (Rafailov et al., 2023):

\[\pi^{*}(y\mid x) = \frac{1}{Z(x)} \pi_{\mathrm{ref}}(y\mid x) \exp\left( \frac{1}{\beta} r^{\mathrm{task}}(x,y) \right), \qquad Z(x) = \sum_{y} \pi_{\mathrm{ref}}(y\mid x) \exp\left( \frac{1}{\beta} r^{\mathrm{task}}(x,y) \right).\]

In other words, reference KL is part of the objective itself, rather than a temporary penalty added only to stabilize training. The optimal policy is the reference policy exponentially tilted by the reward, with $\beta$ controlling the strength of that tilt. When $\beta$ is large, $\pi^{}$ returns toward $\pi_{\mathrm{ref}}$; when $\beta$ is small, $\pi^{}$ concentrates on the highest-reward responses. DPO starts from the same optimum and solves it for the reward expression, avoiding an explicit RL loop.

Once KL is placed in the reward, the RL layer becomes an ordinary maximize-return problem whose optimum is exactly the $\pi^{*}$ above. Keeping KL in the actor loss instead is closer to adding a proximity penalty to each update: it affects only the actor gradient, does not enter the return, and does not pass through credit assignment. Both conventions express reference regularization, yet they optimize different objectives, so the same reference penalty should not be placed in both the reward and actor loss.

When the actor and critic use separate models and optimizers, their losses are backpropagated separately. If they share a backbone, both training signals may update the shared parameters. The reference model, reward model, old log probabilities, advantages, and value targets receive no gradient during the current PPO update.

A complete iteration therefore forms the following loop:

1. Use $\pi_{\mathrm{old}}$ to generate responses for a batch of prompts, and save the sampled tokens, old log probabilities, and old values. Reference-policy log probabilities can be saved during rollout or recomputed by the frozen reference model during the actor update.
2. Compute task rewards, then use GAE to obtain fixed advantages $\hat A_t$ and value targets $\hat R_t$.
3. Split the rollout batch into minibatches. The current actor recomputes probabilities for the sampled tokens and updates $\theta$ through the ratio, clipping, and reference regularization. The current critic recomputes prefix values and updates $\phi$ through the value loss.
4. After a limited number of updates on the current batch, discard the rollouts. The updated policy generates the next batch of responses and becomes the next round’s $\pi_{\mathrm{old}}$.

The core logic of PPO can now be connected. The rollout batch provides old-policy evidence. The probability ratio lets those samples enter the current-policy surrogate and measures how much the sampled actions’ probabilities have changed. Clipping limits how much the same fixed advantages can continue to contribute to optimization. The value model provides a baseline and bootstrap prediction for each prefix, while reference regularization constrains long-term drift across many iterations. After a local update, the policy rolls out again and the entire body of evidence is refreshed.

Group Relative Policy Optimization (GRPO)

The previous section explained how PPO uses an advantage to update the policy, but it also left a practical question: what models does a complete LLM PPO update need in order to obtain those advantages?

Along the policy-optimization path, PPO needs to handle three core models. The policy model is the actor that is actually updated. The value model reads each prefix and predicts the expected return from continuing there. The reference model is usually fixed to the SFT model and provides KL regularization over the course of long-term training. In addition, the system needs a reward model or verifier to provide a task reward for each complete response. The old policy is the policy snapshot used for the current rollout. It is often represented by saved old log probabilities, so keeping another full model is not always necessary.

The value model is an extra burden in PPO. As we saw in the previous section, GAE uses $V_\phi(s_t)$ as a prefix-dependent baseline and constructs value targets $\hat R_t$. Every PPO round therefore updates the policy and also fits the value model to targets obtained from sampled responses. These targets contain sampling noise. After the policy changes, the corresponding prefixes and rewards change as well. In a terminal-reward LLM setting, the critic must also propagate a reward that appears only at the end of the response back to earlier prefixes. This leads to a more specific question: if we have sampled several responses for the same prompt, can we construct an advantage directly from their relative performance and remove the need to train a separate value model?

GRPO starts from this question. Instead of training a value prediction for every prefix, it generates a group of responses for the same prompt and compares their rewards within the group. A response that performs better than the other responses in its group receives a positive update signal, while a worse response receives a negative signal (Shao et al., 2024).

Figure 2 places the two pipelines side by side. Both retain a policy, a reward model or verifier, and a reference policy. The key difference is where the advantage comes from. PPO needs a trained value model, and GAE combines rewards and values into per-token advantages. The implicit returns in GAE also become value targets and train the critic through the value loss, which is the dashed path back to the value model in the figure. GRPO replaces one sample with a group of responses. The group mean acts as the baseline, while the standard deviation only rescales the signal, so the critic and its value loss disappear. In the figure, dashed lines indicate parameter updates. PPO has one path from the advantage back to the policy and another path carrying the value target to the critic. GRPO keeps only the path back to the policy. PPO-style ratios and clipping are omitted because the figure focuses on where the baseline comes from and where KL enters the optimization process.

In the PPO panel, KL follows the reward path. The reference log probability is first subtracted from the task reward to produce a per-token $r_t$, which then enters GAE, so both the advantage and value target include the penalty. In the earlier derivation of the clipped loss, KL was temporarily kept in the actor loss so that task and regularization signals could be viewed separately. It is drawn as a reward term here because that corresponds to the KL-regularized objective with the closed-form optimum described above, and is also common in RLHF implementations. GRPO has no critic, so KL does not need to pass through the reward. It is subtracted after the advantage and during policy optimization, leaving the group mean and standard deviation determined only by the task reward. The two “$-$” nodes in the panels mark the difference between these conventions. The figure uses a maximizing-objective convention, so the GRPO term is $-\beta\,\mathrm{KL}$. In a minimizing loss, it would be $+\beta\,\mathrm{KL}$.

Sampling a Group of Responses from the Same Prompt

Given a prompt $x$, sample $G$ responses from the rollout-time policy $\pi_{\mathrm{old}}$:

\[y_1,y_2,\ldots,y_G \sim \pi_{\mathrm{old}}(\cdot\mid x).\]

The reward model or verifier assigns each response a response-level reward:

\[R_1,R_2,\ldots,R_G.\]

Because the responses share a prompt, they form a natural comparison set. First compute the group’s mean reward:

\[\bar R = \frac{1}{G}\sum_{j=1}^{G}R_j.\]

For response $y_i$, use its difference from the group mean as the relative advantage:

\[\tilde A_i=R_i-\bar R.\]

If $\tilde A_i>0$, the response received a higher reward than the average of this group sampled for the same prompt, so the policy should increase its probability. If $\tilde A_i<0$, it performed worse in this comparison and the policy should lower its probability.

Implementations often also normalize by the group’s standard deviation:

\[\hat A_i = \frac{R_i-\bar R} {\operatorname{std}(R_1,\ldots,R_G)+\epsilon}.\]

This normalization mainly adjusts the scale of the update signal. When rewards differ widely within a group, the update can be stronger. When they are close, the relative advantages become smaller. The group mean provides a prompt-conditioned baseline, but it is different from the value function in PPO. A value function tries to predict future return from any prefix, while the group mean describes the relative average performance of the responses sampled for this prompt in this particular group.

How Relative Advantage Enters the Token-Level Update

GRPO has now produced a response-level advantage. For every sampled token $a_{i,t}$ in response $y_i$, it usually uses the same $\hat A_i$:

\[\hat A_{i,t}=\hat A_i.\]

This is an important difference between GRPO and PPO. Both ultimately compute a policy loss at the token level because a language model produces probabilities one next token at a time. The difference is where the advantage in front of each token comes from and whether it changes with the prefix.

In PPO, GAE and the value model estimate a different $\hat A_t$ for different prefixes. Earlier and later tokens in the same response can therefore receive different advantages because their value predictions differ. GRPO first compares complete responses for the same prompt, then assigns one response’s relative advantage to all of its tokens:

\[-\hat A_i \sum_{t=1}^{T_i} \log\pi_\theta(a_{i,t}\mid s_{i,t}).\]

All tokens in the response therefore share the same $\hat A_i$. The signal does not decide how much an individual token contributed to the final result. It expresses a response-level judgment: for this prompt, did the complete response perform better or worse than the other responses in the group? GRPO removes prefix-level value prediction and the resulting token-level advantage estimation. The loss used to apply the signal is still built from the log probabilities of every token in the response.

GRPO and the PPO Objective

GRPO has now provided a value-model-free $\hat A_i$. We can still use the PPO probability ratio because rollout responses are generated by the old policy, while the current policy performs multiple optimizer steps on the same batch:

\[\rho_{i,t}(\theta) = \frac{\pi_\theta(a_{i,t}\mid s_{i,t})} {\pi_{\mathrm{old}}(a_{i,t}\mid s_{i,t})}.\]

Combining this ratio with the group-relative advantage and retaining PPO clipping gives the core GRPO policy objective:

\[\mathcal L_{\mathrm{GRPO}}(\theta) = -\mathbb E_{i,t} \left[ \min\left( \rho_{i,t}(\theta)\hat A_i, \operatorname{clip} \bigl(\rho_{i,t}(\theta),1-\epsilon,1+\epsilon\bigr)\hat A_i \right) \right].\]

Its structure is the same as the PPO clipped objective. The change lies in the source of the advantage: here $\hat A_i$ comes from the group-relative reward defined above. The ratio still reweights old-policy samples for the current-policy objective, and clipping still limits how far fixed rollout evidence can move the policy.

If training also uses a reference model, GRPO can add the same reference regularization as PPO:

\[\mathcal L_{\mathrm{actor}}(\theta) = \mathcal L_{\mathrm{GRPO}}(\theta) + \beta\mathcal R_{\mathrm{ref}}(\theta).\]

There is no value loss because GRPO does not train a value model and therefore does not need to fit the $\hat R_t$ produced by GAE. A reward model or verifier is still needed, and a reference model may still be present. GRPO removes the critic training path. It does not remove the task reward or reference regularization.

A Complete GRPO Update

A GRPO iteration can be understood in the following order:

1. For each prompt, use the rollout-time policy to sample a group of responses, and save the sampled tokens and old log probabilities.
2. Use a reward model or verifier to compute one complete reward for each response. Then compute the mean, standard deviation, and relative advantages within each group.
3. The current policy recomputes sampled-token probabilities and constructs the actor loss from the probability ratio, group-relative advantage, and PPO clipping. Reference-model regularization can be added if needed.
4. Perform a limited number of optimizer steps on the group samples, then roll out again so that the next batch supplies new group comparisons.

PPO and GRPO both ultimately update token probabilities, but they construct the update coefficient differently. PPO asks: from the current prefix, how much higher is the future expected return than the value model’s prediction? GRPO asks: for the same prompt, how much higher is this response’s reward than the rewards of the other responses in the group? The key simplification in GRPO is that it does not try to perform token-level credit assignment from one response. It first obtains response-level relative performance and turns it into a shared token-level policy signal. PPO’s ratio and clipping then continue to control how this evidence is used.

Starting from GRPO: How Else Can Policy Optimization Be Improved?

GRPO removes the value model and lets the relative rewards of several responses for the same prompt participate directly in the policy update. The training flow becomes lighter. Many choices that were previously handled by the value model and the PPO training loop move into group signals, loss weighting, ratio and clipping, and rollout organization.

Start with GRPO’s own objective. A seemingly simple normalization can change the training weight assigned to different prompts and responses of different lengths. As these choices reveal their limitations, later methods modify GRPO around the specific problems they expose.

Dr. GRPO and DAPO: Group Signal and Objective Weighting

First write down two choices in GRPO that affect the scale of the update. Given rewards $R_1,\ldots,R_G$ for responses to the same prompt, GRPO first subtracts the group mean:

\[\tilde A_i=R_i-\bar R, \qquad \bar R=\frac{1}{G}\sum_{j=1}^{G}R_j.\]

A common GRPO formulation then normalizes the result by the group standard deviation:

\[\hat A_i=\frac{\tilde A_i}{\operatorname{std}(R_1,\ldots,R_G)+\epsilon}.\]

Subtracting the group mean provides a baseline. It answers how much better this response performed than the group average, which is exactly the comparison GRPO wants. Dividing by the standard deviation does something else: it rescales all updates from a prompt according to that group’s reward spread. As a result, the update strength for different prompts depends on the composition of their sampled groups.

Consider a binary-reward example. The first group’s rewards are $(1,1,0,0)$. Its mean is $0.5$ and its population standard deviation is $0.5$, so successful and failed responses receive advantages of $+1$ and $-1$. The second group’s rewards are $(1,0,0,0)$. Its mean is $0.25$ and its standard deviation is approximately $0.433$. The successful response receives an advantage of about $+1.73$, while the failed responses receive about $-0.58$. Both groups are making the same comparison between successful and failed responses, yet after group normalization the update scale of an individual response is different.

This scaling is not necessarily unreasonable. It emphasizes rare outcomes, which can help the policy focus on an unusual correct response or an unusual mistake. The problem is that the amplification comes only from the reward spread of the current group. Standard deviation cannot tell whether a rare outcome is valuable exploration or a sampling fluctuation.

If each prompt should contribute according to the original data distribution, per-group standard deviation introduces an additional prompt weighting: groups with a lower standard deviation receive larger advantage coefficients. This extra reweighting does not come from the original expected-reward objective and therefore deserves separate scrutiny. Removing it reduces prompt-specific scaling while retaining the group mean as the baseline.

GRPO also has an independent length-normalization choice. Let $\ell_{i,t}(\theta)$ denote the token-level term formed from the ratio, clipping, and advantage. A common response-level aggregation first averages the token loss within each response and then averages over the responses in a batch:

\[\mathcal L_{\mathrm{response}}(\theta) =- \frac{1}{G} \sum_{i=1}^{G} \frac{1}{T_i} \sum_{t=1}^{T_i} \ell_{i,t}(\theta).\]

This gives every response the same total weight in the batch. Suppose one response contains $100$ tokens and another contains $1000$. Each token in the shorter response receives a coefficient of $1/100$, while each token in the longer response receives $1/1000$. An individual token in the long response therefore has one tenth the update signal of an individual token in the short response. There is no algebraic error here. It expresses the choice to give each response equal weight as a whole.

The issue is that a terminal reward is copied to every token in the response. For a long response with a positive advantage, many reasoning tokens share one signal that has been reduced by $1/T_i$. For a long response with a negative advantage, the penalty on each token is weakened in the same way. Response length therefore affects how a reward is converted into token updates. For long CoT, this weighting choice can create a response-level length bias.

Dr. GRPO, short for Group Relative Policy Optimization Done Right, changes both normalizations above (Liu et al., 2025). For the group-level signal, it retains the centered reward:

\[\hat A_i^{\mathrm{Dr}}=R_i-\bar R.\]

Dr. GRPO omits the group standard deviation because it adds an extra scale to each prompt that is determined by the sampled composition of the current group. If the rewards in one group happen to be close, the standard deviation is small and a small reward difference is amplified. If the rewards in another group are more spread out, the same reward difference is reduced. The update strength of one response is therefore affected by the sampling outcomes of the other responses in its group.

This scaling can help emphasize rare outcomes, yet it can also mix prompt difficulty and sampling noise into the policy update. Dr. GRPO keeps the relative comparison supplied by the group mean and removes this extra prompt-dependent scaling, so the reward difference itself determines the update signal.

It also removes the per-response $1/T_i$ normalization and uses one fixed global denominator to aggregate all token contributions. Response length no longer changes the coefficient of an individual token through the response’s own token count. Using $\ell_{i,t}(\theta)$ for the token-level clipped surrogate, the core of Dr. GRPO can be written as

\[\mathcal L_{\mathrm{Dr.GRPO}}(\theta) \propto -\sum_i\sum_{t=1}^{T_i} \ell_{i,t}(\theta;\hat A_i^{\mathrm{Dr}}),\]

where the fixed loss denominator is omitted. The key point is that it does not use the current response’s token count.

Dr. GRPO therefore focuses on how an existing group-relative signal enters the objective. It keeps the group comparison and removes group standard deviation and per-response length normalization, so prompt difficulty and response length do not change the update strength through extra weighting.

This weighting correction still assumes that reward varies within a group. DAPO, short for Decoupled Clip and Dynamic Sampling Policy Optimization, retains the basic group-relative advantage, including group standard deviation (Yu et al., 2025):

\[\hat A_i^{\mathrm{DAPO}} = \frac{R_i-\bar R} {\operatorname{std}(R_1,\ldots,R_G)+\epsilon}.\]

It addresses a different situation: some groups have no relative signal at all. For a binary reward, suppose four responses are sampled for one prompt. If they are all correct, the rewards are

\[(1,1,1,1),\]

the group mean is $1$, and every centered reward is $0$. If they are all wrong, the rewards are

\[(0,0,0,0),\]

the group mean is $0$, and the centered rewards are again all $0$. This is perfectly reasonable: neither group contains information for comparison. The all-correct group does not tell us which response is better, and the all-wrong group does not tell us which response deserves a higher probability. A zero policy gradient from such a group is the natural result of this estimator.

The training-efficiency problem is that as the policy improves, more prompts may produce all-correct groups. For prompts that remain difficult, all-wrong groups may persist. Every rollout still has to generate and score these responses, yet they provide no direction for a group-relative update. As fewer prompts in the batch provide a signal, the gradient becomes weaker and more exposed to the noise in the remaining samples.

DAPO’s Dynamic Sampling adds the following condition for binary-reward groups:

\[0<\sum_{i=1}^{G}R_i<G.\]

The condition says that a group must contain at least one successful and at least one failed response. DAPO filters out groups with accuracy $0$ or $1$ and continues sampling until the batch contains enough mixed groups. It does not claim that an all-correct group has a hidden token-level gradient. It avoids spending most of the computation on samples from which the current group-relative estimator cannot obtain a direction.

DAPO also addresses three additional issues in long-CoT training.

The first concerns clipping. Standard PPO-style clipping uses the symmetric interval

\[[1-\epsilon,\,1+\epsilon].\]

For a positive advantage, the policy tries to raise the sampled token’s probability. Once the ratio exceeds the upper bound, the clipped objective no longer rewards further increases. Consider a concrete example. Suppose a token’s probability during rollout is only $0.01$ and PPO’s upper clip is $1.2$. If we temporarily interpret this ratio boundary as a hard constraint, the token’s probability can be pushed only to

\[0.01\times 1.2=0.012.\]

Its absolute increase is only $0.002$. A token that was already common has a larger absolute probability change under the same relative ratio. It is also important that PPO clipping constrains the surrogate objective rather than strictly guaranteeing that the current policy probability never exceeds $0.012$. Its direct effect is that once the ratio exceeds $1.2$, the sample stops providing additional optimization benefit in the positive-advantage direction.

This explains why the upper clip can limit exploration. A low-probability token may receive a positive advantage from a small number of successful rollouts, yet enter the clipped region before it has time to grow from $0.01$ to a probability at which it is sampled more easily. The same evidence then has little ability to keep pushing it upward. DAPO calls its change Clip-Higher. It decouples the lower and upper clips and sets a wider upper bound. The policy still limits excessively large updates, while low-probability exploration tokens get more room to grow.

The second issue is loss aggregation. The Dr. GRPO section showed that after removing per-response $1/T_i$, the objective can directly accumulate all active tokens. DAPO makes a similar choice and calls it the Token-Level Policy Gradient Loss:

\[\mathcal L_{\mathrm{token}}(\theta) =- \frac{\sum_i\sum_{t=1}^{T_i}\ell_{i,t}(\theta)} {\sum_i T_i}.\]

In DAPO, the denominator is the number of active tokens in the current batch, $\sum_iT_i$. Every active token in the batch therefore has the same coefficient, and a long response has a larger total contribution because it contains more tokens. Dr. GRPO uses a fixed global denominator, while DAPO uses a batch-dependent denominator. The two make the same choice at the token-aggregation level. Their main difference is the overall gradient scale.

The third issue is reward treatment for overlong responses. Long-CoT training usually sets a maximum generation length. If a response is truncated when the budget runs out, the verifier may see only an incomplete answer. A failure reward cannot distinguish an incorrect reasoning process from a response that simply needed a few more tokens. If every truncated response is treated as an ordinary failure, the policy may suppress a reasoning trajectory that would have been useful.

DAPO offers two treatments. The first is Overlong Filtering: mask the policy loss for truncated responses so that they do not participate in the current update. This acknowledges that the final reward for this response is insufficient to judge whether its reasoning was effective, so the evidence is temporarily unused.

The second is Soft Overlong Punishment. It places a buffer region before the maximum length. A response outside the region receives no length penalty. Once it enters the buffer, the penalty grows as it approaches the maximum, and exceeding the maximum receives the largest penalty. Let $L_{\max}$ be the maximum length and $L_{\mathrm{cache}}$ the buffer length. One piecewise form is

\[R_{\mathrm{len}}(y)= \begin{cases} 0, & |y|\leq L_{\max}-L_{\mathrm{cache}},\\ \dfrac{(L_{\max}-L_{\mathrm{cache}})-|y|}{L_{\mathrm{cache}}}, & L_{\max}-L_{\mathrm{cache}}<|y|\leq L_{\max},\\ -1, & |y|>L_{\max}. \end{cases}\]

This reward shaping gives the policy a gradually strengthening signal. The model can continue longer reasoning, but as it approaches the generation budget it needs to learn to finish earlier. Compared with marking every overlong response as a failure, this reduces reward noise from truncation and turns the length budget into a more continuous training signal.

Dr. GRPO and DAPO address adjacent problems. Dr. GRPO revisits how the GRPO objective assigns weight to responses and tokens. DAPO goes further into the exploration, effective sampling, loss aggregation, and overlong-response issues of long-CoT training. Both continue to use group-relative advantages and neither needs a separately trained value model. Their main changes occur after the advantage is formed and in how rollout data enters the training loop.

This leaves a concrete question. GRPO’s reward and advantage are defined over a complete response, while its ratio and clipping still operate on individual tokens. Does a response-level reward have to be paired with token-level policy correction in this way? If we treat the complete response as one sampling event, should importance correction also happen at that level?

GSPO: Aligning the Importance Ratio with the Response

GRPO and DAPO both obtain one relative advantage from a complete response and then expand the policy loss over that response’s tokens. It helps to distinguish two levels here: how a response-level reward reaches an autoregressive policy, and how the probability mismatch between the old and current policies is corrected. The first is natural because a response’s log probability decomposes as

\[\log\pi_\theta(y_i\mid x) = \sum_{t=1}^{T_i} \log\pi_\theta(y_{i,t}\mid s_{i,t}).\]

If $\hat A_i$ is the advantage of the complete response, then

\[\hat A_i \sum_{t=1}^{T_i} \nabla_\theta\log\pi_\theta(y_{i,t}\mid s_{i,t})\]

is how a response-level reward enters the token-level policy gradient. Sharing the response advantage across its tokens follows from the chain-rule decomposition of response probability. The level that needs checking is the other one: should the importance ratio also be computed token by token?

The importance ratio is the part that needs re-examination. In its token-level objective, GRPO uses

\[\rho_{i,t}(\theta) = \frac{\pi_\theta(y_{i,t}\mid s_{i,t})} {\pi_{\mathrm{old}}(y_{i,t}\mid s_{i,t})}.\]

If clipping is omitted, one response contributes to the gradient as

\[g_i^{\mathrm{GRPO}} \propto \hat A_i \sum_{t=1}^{T_i} \rho_{i,t}(\theta) \nabla_\theta \log\pi_\theta(y_{i,t}\mid s_{i,t}).\]

For a short response, these token-level ratios may be relatively easy to control. As the response becomes longer, many sampled-token ratios enter the update together and their local fluctuations accumulate. A sequence-level reward paired with token-level importance correction can then produce a high-variance gradient. Intuitively, the training signal evaluates the outcome of the complete response, while importance correction processes that response as many local probability changes. The longer the response, the more opportunities there are for noise in an individual token ratio to affect the update for the whole response.

If the complete response is treated as the object of importance sampling, its likelihood ratio should be built from the likelihood of the full response:

\[\frac{\pi_\theta(y_i\mid x)} {\pi_{\mathrm{old}}(y_i\mid x)} = \prod_{t=1}^{T_i} \rho_{i,t}(\theta).\]

Group Sequence Policy Optimization (GSPO) redefines the importance ratio from this perspective. It uses a length-normalized geometric mean (Zheng et al., 2025):

\[s_i(\theta) = \left( \prod_{t=1}^{T_i}\rho_{i,t}(\theta) \right)^{1/T_i} = \exp\left( \frac{1}{T_i} \sum_{t=1}^{T_i}\log\rho_{i,t}(\theta) \right).\]

The averaging happens over log ratios, so this is a geometric mean rather than an arithmetic mean. $s_i$ gives the complete response one shared policy-change coefficient and avoids making a long response produce an extreme ratio merely because it contains more tokens. Response-level reward, importance ratio, and clipping now operate on the same response-level object.

GSPO can therefore write a GRPO-style objective with sequence-level clipping:

\[\mathcal L_{\mathrm{GSPO}} = -\frac{1}{G} \sum_{i=1}^{G} \min\left( s_i\hat A_i,\, \operatorname{clip} \left( s_i,1-\epsilon,1+\epsilon \right)\hat A_i \right).\]

The $\hat A_i$ still comes from the relative reward of the complete response, and it still propagates to the log-probability gradients of every token in that response. The change is the unit used for the ratio, clipping, and objective aggregation. GRPO uses a different $\rho_{i,t}$ for each token. GSPO first computes one shared $s_i$ for the sequence and clips that sequence-level quantity. GSPO changes the overall reweighting of old rollout evidence. The way the response-level advantage enters the token-level gradient still comes from the same log-probability decomposition.

In agentic search, Dynamic-filter Sequence-level Policy Optimization (DSPO) combines GSPO’s sequence-level optimization with DAPO-style dynamic filtering. The former addresses update stability over long trajectories, while the latter filters all-success and all-failure groups so that the group-relative signal remains useful (Gu et al., 2025). It is best viewed as a combination recipe for agentic RL, so we will not expand it separately here.

The sequence-level ratio resolves the mismatch in granularity between the reward and importance correction. The clipping rule itself still uses a hard boundary, though. Is this boundary the best way to control fixed rollout evidence?

CISPO and SAPO: Rethinking Clipping

First recall the PPO actor loss. This is the part CISPO changes. We will write only the actor loss for now, since the value loss and reference-model regularization do not affect this comparison:

\[\mathcal L_{\mathrm{PPO}} = -\mathbb E_t\left[ \min\left( \rho_t\hat A_t, \operatorname{clip}(\rho_t,1-\epsilon,1+\epsilon)\hat A_t \right) \right],\]

where

\[\rho_t = \frac{\pi_\theta(a_t\mid s_t)} {\pi_{\mathrm{old}}(a_t\mid s_t)}.\]

$\hat A_t$ says whether this sample is worth learning from and in which direction. $\rho_t$ says how the current policy has changed the probability of this sampled action relative to the rollout-time policy. PPO compares the original importance-weighted term with the clipped term, and $\min$ returns the more conservative value for the policy update.

As Figure 1 shows, once PPO enters the clipped region it returns a fixed objective term. The term still has a value, yet it no longer changes with the current policy, so the sample’s policy gradient is $0$. CISPO addresses exactly this truncated gradient path.

Consider first the case $\hat A_t>0$. This token performed better than expected under the current prefix, so the policy tends to increase its probability. The original term $\rho_t\hat A_t$ keeps increasing with $\rho_t$. To limit how much the same old rollout evidence can push the policy, PPO returns the clipped term when $\rho_t>1+\epsilon$:

\[\min\left( \rho_t\hat A_t, \operatorname{clip}(\rho_t,1-\epsilon,1+\epsilon)\hat A_t \right) = (1+\epsilon)\hat A_t.\]

The trade-off is clearer on the scale of token probabilities. Suppose a token had probability $0.01$ during rollout. Its response receives a positive advantage, and after one update the current policy raises the probability to $0.013$, so

\[\rho_t=\frac{0.013}{0.01}=1.3.\]

With $\epsilon=0.2$, $\rho_t$ is already above $1+\epsilon=1.2$. If $\hat A_t=2$, PPO returns the objective term $(1+\epsilon)\hat A_t=2.4$. The term still has a value, and the actor loss is still recorded as $-2.4$, but it no longer changes with the current policy, so this sample’s gradient is $0$. This refers to the contribution of this sample. Other samples in the batch can still produce gradients.

PPO’s choice has a clear conservative intuition. Once the ratio crosses the boundary, the current policy has changed substantially relative to the rollout-time policy, and repeatedly reusing the same old evidence might make the update unstable. CISPO focuses on the other side of the trade-off. The token’s absolute probability has increased only from $0.01$ to $0.013$, so it is still rarely sampled. If the response represents a rare but valuable reasoning path, PPO has completely stopped obtaining new positive signal from it. For a sample with $\hat A_t<0$, the situation is symmetric: when $\rho_t<1-\epsilon$, PPO returns $(1-\epsilon)\hat A_t$ and likewise stops amplifying that direction.

Clipped IS-weight Policy Optimization (CISPO) starts from this observation. It keeps an upper bound on the importance weight while keeping the log probability differentiable. Specifically, it clips the ratio into a bounded coefficient, treats that coefficient as a stop-gradient weight, and uses a REINFORCE-style log-probability loss (MiniMax, 2025):

\[\mathcal L_{\mathrm{CISPO}} = -\mathbb E_{i,t} \left[ \operatorname{sg}\left( \operatorname{clip} \left( \rho_{i,t}, 1-\epsilon_{\mathrm{low}}, 1+\epsilon_{\mathrm{high}} \right) \right) \hat A_i \log\pi_\theta(a_{i,t}\mid s_{i,t}) \right].\]

Here $\operatorname{sg}$ denotes stop-gradient. We use GRPO’s notation: $\hat A_i$ is the advantage of response $i$, and all tokens in that response share this signal. $t$ indexes the token. CISPO changes how the ratio enters the gradient. The clipped ratio serves only as a bounded update coefficient, while $\log\pi_\theta$ explicitly retains a differentiable path through the current policy. Let

\[\bar\rho_{i,t} = \operatorname{clip} \left( \rho_{i,t}, 1-\epsilon_{\mathrm{low}}, 1+\epsilon_{\mathrm{high}} \right).\]

Then the gradient contribution of CISPO can be written directly as

\[\nabla_\theta \mathcal L_{\mathrm{CISPO},i,t} = -\bar\rho_{i,t}\hat A_i \nabla_\theta\log\pi_\theta(a_{i,t}\mid s_{i,t}).\]

Using the numerical example above, with $\hat A_i=2$, $\rho_{i,t}=1.3$, and an upper clip of $1.2$, we have $\bar\rho_{i,t}=1.2$. The gradient coefficient in CISPO is $1.2\times2=2.4$, so the sample still updates the policy. Even if $\hat A_i=100$, the coefficient becomes $1.2\times100=120$, so the advantage still affects the update magnitude. CISPO leaves the advantage unchanged. The clipped object is always the importance ratio that reflects policy mismatch.

Why does CISPO write $\log\pi_\theta$ explicitly when $\pi_\theta$ already appears inside the ratio? In PPO, when $\min$ returns the original term, we can indeed obtain the log-probability gradient through

\[\nabla_\theta\rho_t = \rho_t\nabla_\theta\log\pi_\theta.\]

When $\min$ returns the clipped term, $\rho_t$ has been replaced by a constant, so this path is cut off. CISPO treats the clipped ratio as a detached coefficient and keeps $\log\pi_\theta$ in the loss, preserving the gradient path.

This is why the two formulas look similar while their training behavior differs. In the two clipped cases of PPO in Figure 1,

\[\nabla_\theta \mathcal L_{\mathrm{PPO},t}=0,\]

whereas CISPO retains

\[\nabla_\theta \mathcal L_{\mathrm{CISPO},i,t} = -\operatorname{sg}(\bar\rho_{i,t})\hat A_i \nabla_\theta\log\pi_\theta(a_{i,t}\mid s_{i,t}).\]

The treatment of one piece of old evidence can therefore be summarized as follows: CISPO limits its importance weight while still allowing the evidence to produce a gradient through the log probability. This preserves the learning signal from rare but valuable tokens without letting the ratio amplify the update without bound.

CISPO has now fixed the problem that PPO loses the gradient in clipped cases. Looking back at Figure 1, once PPO enters a clipped case, the selected objective no longer changes with the current policy and the individual sample’s gradient is $0$. CISPO uses the clipped ratio as a bounded coefficient while retaining the log-probability gradient.

To place the three policy-gradient behaviors on a common scale, let $c_t$ denote the coefficient in the policy gradient:

\[\nabla_\theta \mathcal L_t = -c_t\hat A_t \nabla_\theta\log\pi_\theta(a_t\mid s_t).\]

For the case $\hat A_t>0$, Figure 3 uses the same $\rho_t$ horizontal axis to plot how the three methods treat the policy-gradient coefficient. Figure 1’s vertical axis represents the per-token surrogate objective. Figure 3’s vertical axis represents the policy-gradient coefficient.

Look first at the left and middle panels of Figure 3. PPO becomes $0$ after the favorable clipping boundary, while CISPO retains a gradient and fixes the coefficient at a plateau. This is CISPO’s improvement over PPO: a clipped sample can still influence the policy update.

The middle panel also reveals a problem that CISPO leaves behind: it still uses hard clipping. With an upper clip of $1.2$ and a positive advantage, CISPO’s coefficient is $c_t=\min(\rho_t,1.2)$. At $\rho_t=1.19$, the coefficient is $1.19$. At $\rho_t=1.21$, it suddenly becomes $1.2$. When $\rho_t$ increases to $2.0$, the coefficient is still $1.2$.

There is first a continuity issue: immediately after the ratio crosses the clipping boundary, the coefficient suddenly enters a plateau. More importantly, $\rho_t=1.21$ and $\rho_t=2.0$ represent different degrees of relative probability change under the current policy, yet CISPO assigns them the same coefficient. The farther the ratio moves from $1$, the farther the sampled action has moved from the rollout policy. CISPO no longer distinguishes those degrees of distance. We want to retain CISPO’s gradient while making the coefficient decrease continuously as the ratio moves farther away.

This is the problem Soft Adaptive Policy Optimization (SAPO) addresses. It replaces the hard boundary with a smooth gate. When the ratio is close to the on-policy point $\rho_t=1$, most of the update is retained. As the ratio gradually moves away from $1$, the coefficient gradually decays. A token that has just crossed the old clipping boundary can still provide a relatively strong signal, while a token that is far from the old policy receives stronger suppression (Gao et al., 2025).

SAPO uses a smooth, temperature-controlled gate centered at the on-policy point $\rho=1$, so the update coefficient decays as the ratio moves away from $1$. Its formal form is

\[w_{i,t}(\theta) = 4p_{i,t}(\theta)\bigl(1-p_{i,t}(\theta)\bigr), \qquad p_{i,t}(\theta) = \sigma\left( \tau_{i,t}\bigl(\rho_{i,t}(\theta)-1\bigr) \right),\]

where $\sigma$ is a sigmoid and $\tau_{i,t}$ is the temperature. $w_{i,t}$ is the gate. The actual policy-gradient coefficient also includes the importance ratio, so it has the form $w_{i,t}\rho_{i,t}\hat A_{i,t}$. When $\rho_{i,t}=1$, $w_{i,t}=1$ and the full update is retained. As the ratio moves away from $1$, $w_{i,t}$ decreases smoothly. In the right panel of Figure 3, the three SAPO curves use $\tau=2,5,10$: a larger $\tau$ produces faster decay, while a smaller $\tau$ keeps the gate closer to the unclipped importance weighting. SAPO also uses different temperatures for positive and negative advantages, usually applying faster decay to negative-token updates to suppress the high-variance gradients they can spread across a large vocabulary.

The three designs can be understood as responses to the same problem. PPO and GRPO directly truncate the surrogate with a hard clip. CISPO treats the clipped ratio as a bounded importance weight and keeps the gradient path through the log-probability loss. SAPO uses a continuous gate so that the update decays gradually with the ratio’s distance. All three address the limited reliability of fixed rollout evidence, while making different choices about what should happen to the gradient beyond that range.

DRPO and VPO: Rethinking the Reward Signal

The methods so far have assumed that the reward has already been organized into a signal usable by the policy update. This section moves the question one step earlier: how should the signal express the goals that actually matter for the task? Decoupled Reward Policy Optimization (DRPO) considers the case where correctness and reasoning length are mixed into one scalar reward (Li et al., 2025). Vector Policy Optimization (VPO) considers the case where test-time search needs a diverse set of candidates, while the training objective encourages only one scalar optimum (Bahlous-Boldi et al., 2026).

Start with DRPO. Consider a mathematical reasoning task. The most basic correctness reward only checks whether the answer is correct: a correct answer receives $1$, and an incorrect answer receives $0$. To reduce overthinking, training often adds a length reward as well. When correctness is the same, shorter reasoning receives a higher reward. For example, a correct response can receive a length-dependent reward $r_{\mathrm{len}}(o)$ that decreases with its length, which is then combined with the correctness reward. Six responses sampled for one prompt might have the following correctness values:

\[(1,1,1,0,0,0),\]

After adding the length reward, their final rewards might be

\[(0.73,0.60,0.20,0,0,0).\]

These numbers express a reasonable preference. The first three responses are correct, but the third is longer and therefore receives a lower reward. The other three are incorrect and remain at $0$. The problem appears in the GRPO group-relative advantage:

\[\hat A_i=R_i-\bar R.\]

The group mean is $0.255$, so the advantage of the third response is

\[0.20-0.255<0.\]

This response gave the correct answer, yet because it is longer than the other two correct responses, it falls below the group mean. Its policy-update direction becomes negative, and the policy lowers the probability of the tokens in this correct reasoning trajectory. The length reward was meant to express an efficiency difference among correct answers. After group normalization, it can change the direction of the correctness signal.

This leaves a meaningful design question: how should correctness and length enter the learning signal together? We want incorrect responses to receive a negative constraint from correctness. Among correct responses, we want to compare efficiency, giving a shorter correct response a stronger positive signal and a longer correct response a weaker positive signal. A longer correct answer should be downweighted without being treated as an incorrect answer.

Decoupled Reward Policy Optimization (DRPO) separates these comparisons. It first divides responses into positive and negative groups according to correctness, then uses the length reward to redistribute weights only within the positive responses. Length preference affects the relative strengths among correct responses, while correctness continues to distinguish correct from incorrect responses. One concrete implementation can use length-based weights normalized only within the positive group, while the subsequent policy optimization reuses the existing framework.

Another source of room for improvement comes from test-time search. Suppose we train a code-generation model and evaluate each response on two test cases, so the reward is a two-dimensional vector:

\[\mathbf r(y_A)=(1,0), \qquad \mathbf r(y_B)=(0,1).\]

$y_A$ is strong on the first type of test case, while $y_B$ is strong on the second. If training always uses $w^{*}=(0.5,0.5)$ to compress the vector into a scalar reward, both responses score $0.5$. Another response $y_C$ with reward vector $(0.6,0.6)$ receives scalar reward $0.6$, so training favors responses like $y_C$.

If we ultimately need only one response, this result is not obviously problematic. In test-time search, however, we usually sample several candidates and choose among them for the current task. Future search may emphasize the first type of test case or the second. When $w=(0.9,0.1)$, $y_A$ scores $0.9$. When $w=(0.1,0.9)$, $y_B$ scores $0.9$. $y_C$ scores only $0.6$ under both preferences. If training pushes every sample toward $y_C$, additional sampling budget produces many similar answers and search loses the opportunity to choose among different strategies.

Vector Policy Optimization (VPO) changes the goal from finding one response with the highest scalar reward to generating a set of responses that covers different reward trade-offs. For a prompt, the model produces a response set $S$, and each response has its own reward vector. During training, the scalarization weight changes, so different responses become the best candidate under different weights.

Formally, treat each weight vector $w$ as a possible downstream preference and score the set with

\[R(S) = \mathbb E_{w} \left[ \max_{y\in S}w^{\top}\mathbf r(x,y) \right].\]

The inner $\max$ represents the current preference selecting the most suitable candidate from the set. The outer expectation means that training wants the set to contain a useful answer for many preferences. In practice, the method samples weight vectors satisfying $w_j\geq0$ and $\sum_jw_j=1$, for example from a Dirichlet distribution.

VPO therefore changes the learning signal used by GRPO. It first computes an advantage from the set-level reward and then uses the existing policy-gradient machinery to update the tokens that generated the responses. What needs to be preserved is diversity in reward space. If two responses perform identically across every reward dimension, they offer no new choice for test-time search even if their surface text differs.

DRPO and VPO both reconstruct the learning signal before the policy gradient, yet they answer different questions. DRPO handles the interference between correctness and length in a composite reward. VPO handles candidate collapse caused by scalar RL and places the reward-space coverage needed by test-time search into the training objective. They operate at a different level from CISPO and SAPO, which ask how to control the update after evidence has been obtained.

SAO: Asynchronous Agentic RL

So far, GRPO, DAPO, and GSPO have all assumed a relatively synchronous training rhythm: sample a group of responses for each prompt, wait for those responses to finish and receive rewards, and then update from the rollout batch. This structure is natural when response lengths are similar. In agentic RL, a trajectory may contain dozens or hundreds of tool interactions, and completion times can diverge sharply. A short trajectory may already be finished while the system still waits for the slowest trajectory in its group, leaving rollout workers and the learner idle.

SAO, or Single-Rollout Asynchronous Optimization, starts from this synchronization problem (Hou et al., 2026). It no longer requires a full set of trajectories for the same prompt to arrive together. Each prompt produces only one rollout, and a trajectory can enter the learner as soon as it finishes.

This choice first removes GRPO’s most important baseline. GRPO’s relative advantage comes from comparing multiple responses for the same prompt:

\[\hat A_i = R_i-\frac{1}{G}\sum_{j=1}^{G}R_j.\]

If each prompt has only one rollout, $G=1$, so

\[\hat A_i=R_i-R_i=0.\]

Single-rollout training therefore cannot directly reuse the group-relative advantage. SAO reintroduces a PPO-style value model. For a state $s_t$ in the trajectory, the critic $V_\phi(s_t)$ estimates the expected return from continuing there and uses it to construct a token-level advantage. As we saw in PPO, the one-step TD error is

\[\delta_t = r_t+\gamma V_\phi(s_{t+1})-V_\phi(s_t),\]

followed by GAE, which combines estimates at different horizons. The baseline now comes from a learned value function rather than from other sampled trajectories for the same prompt. Single rollout reduces group synchronization, while bringing back the critic that GRPO had avoided. If the value estimate is inaccurate, advantage variance becomes a problem again.

The feature that most clearly separates SAO from ordinary PPO is asynchronous rollout. In synchronous PPO, one rollout round is usually produced by one frozen old policy, and the optimizer begins only after the complete batch has been collected. The behavior probability of a sampled token therefore has a clear source:

\[\rho_t(\theta) = \frac{\pi_\theta(a_t\mid s_t)} {\pi_{\mathrm{old}}(a_t\mid s_t)}.\]

Asynchronous training removes this clean time boundary. Rollout generation and optimizer updates can proceed at the same time. A coding trajectory may spend a long time waiting for a shell command, compilation, or tests while the learner updates several times from other trajectories that have already finished. SAO considers an even more aggressive system in which the rollout engine can synchronize new weights during the generation of the same trajectory, so earlier and later model calls can come from different rollout-model versions.

For example, one trajectory may pass through

\[\pi_1 \longrightarrow \text{tool execution} \longrightarrow \pi_2 \longrightarrow \text{tests} \longrightarrow \pi_4.\]

There is no single historical checkpoint that can simply represent the behavior policy for the entire trajectory.

SAO continues to use the same probability ratio. The $t$-th sampled token came from the policy used by the rollout engine at that moment, so the ratio is

\[\rho_t(\theta) = \frac{\pi_\theta(a_t\mid s_t)} {\pi_{\mathrm{rollout},t}(a_t\mid s_t)}.\]

The subscript $t$ is kept because different model calls in one long trajectory may be produced by different rollout-model versions. Each token should use the behavior policy that actually generated it.

An importance ratio alone is insufficient. If a rollout is already very stale, the ratio between the current and behavior policies may be far from $1$, making that evidence increasingly questionable as a guide for the current policy. SAO therefore uses Direct Double-Sided Importance Sampling and keeps a token only when its ratio lies inside a specified interval:

\[f(\rho_t;\epsilon_l,\epsilon_h) = \begin{cases} \rho_t, & 1-\epsilon_l<\rho_t<1+\epsilon_h,\\ 0, & \text{otherwise}. \end{cases}\]

This differs from ordinary PPO clipping in an important way. PPO usually clips an excessive ratio to the boundary. SAO directly masks a token whose ratio has moved outside the trust region, so it no longer produces a policy gradient. It is willing to discard some stale evidence rather than let tokens with unreliable local information push the current policy in a direction too far from the behavior policy.

Single rollout also makes critic quality more important. In its training setup, SAO updates the value model more frequently than the actor and freezes attention parameters during critic training, updating only selected projection parameters. Their ablations suggest that these choices help the value model keep up with the changing policy and reduce instability in the critic itself.

Multi-turn agents introduce a problem absent from an ordinary text response. A trajectory often alternates between

\[[a_0,o_0,a_1,o_1,\ldots],\]

where $a_i$ is an action generated by the policy and $o_i$ is an observation returned by the environment. Observations are not action tokens sampled by the model. Taking adjacent value differences over the entire token sequence would propagate environment feedback as if it were part of the model’s action sequence. SAO therefore uses skip-observation token-level GAE, connecting advantage estimation from one model action to the next while skipping the intervening observation tokens.

Taken together, these choices mean that SAO does more than replace PPO’s clipping rule. It changes the rollout unit first: instead of waiting for a complete group for one prompt, it lets each completed trajectory enter the learner quickly. This removes the group barrier, loses GRPO’s relative baseline, and makes rollout generation genuinely overlap with policy updates. The value model, rollout-time log probabilities, strict token masking, and skip-observation GAE all follow from this asynchronous single-rollout setting.

At this point, policy optimization can no longer be cleanly separated from rollout architecture. For a long-horizon agent, when a trajectory finishes, how long it waits before being used, and whether the policy has changed during generation all begin to determine which evidence the optimizer can trust.

From Policy Objective to Training Loop: Practical Problems in Agentic RL

The preceding SAO section gave us a concrete signal: in long-horizon agentic RL, the policy objective can no longer be designed independently of the rollout architecture. Once grouped synchronous rollout is replaced with single-rollout asynchronous training, GRPO’s relative baseline disappears, and the critic, GAE, rollout-time probabilities, and stale-sample handling all need to return to the training loop.

There is a more general issue behind this. So far, we have treated a rollout as evidence for the optimizer: the policy generates a response, a verifier returns a reward, and the advantage determines how the probabilities of sampled actions should change. For mathematical reasoning or single-turn code generation, this abstraction is relatively clean. The model generates a response, the episode ends, and the environment has little persistent state of its own.

Agentic RL stretches that evidence into a real trajectory. The model may search code, read files, modify an implementation, run tests, and continue acting from a new error before finally submitting a result:

\[\tau=(s_0,a_0,o_1,s_1,a_1,o_2,\ldots,s_T).\]

An action may be natural language, a shell command, a file edit, or a structured tool call. The observation comes from an environment that previous actions have already changed. Software-engineering agents are a typical example. Nebius describes this as a multi-turn RL problem with a stateful environment and non-trivial feedback, clearly different from a single-turn reasoning task without intermediate environment feedback (Golubev et al., 2025).

Once the trajectory becomes a closed loop, more factors determine the learning signal that the optimizer receives. Is the environment stable? Which actions should receive credit for the final reward? Did the rollout framework preserve the trajectory faithfully? How much has the policy changed by the time the trajectory reaches the learner? Each question affects whether a policy gradient that is formally valid still deserves to be trusted.

How a Rollout Is Produced

In mathematical RL, the environment can sometimes be almost invisible. The model generates an answer, an exact-match checker returns $0$ or $1$, and the rollout ends.

A coding agent usually needs a complete software environment to compute its reward. The task requires a fixed repository snapshot, dependencies, a shell, a test suite, and a filesystem. Each agent action changes that environment, which changes later observations. If two concurrent rollouts share a mutable directory, they can even modify each other’s code directly.

An actual SWE rollout therefore often has to create an isolated environment, launch an agent harness, execute a sequence of tool calls, save the actions and observations, and finally run tests to obtain a reward. R2E-Gym, used by DeepSWE, defines Bash, Search, File Editor, and Finish/Submit as its action space and produces a sparse outcome reward based on whether the modified code passes the tests. Its training report also records a concrete scaling problem: one RL iteration required launching 512 Docker containers in parallel. As the scale increased, the Docker API server could overwhelm dockerd, so the system eventually had to use Kubernetes to schedule the environments (Luo et al., 2025).

The important point is that rollout throughput is no longer determined by the decoder alone. CPUs, container startup, filesystem I/O, package installation, test execution, the scheduler, and tool latency can all become bottlenecks. A GPU may generate the next shell command quickly and then wait several minutes for compilation. While one trajectory waits for tests, another may complete a dozen model calls.

NVIDIA’s NeMo Gym SWE case study gives another intuitive set of numbers: 16 prompts with 32 rollouts per prompt means roughly 512 task instances running concurrently, which roughly requires 512 CPU cores. In their setup at the time, the fastest SWE RL step with a batch size of 512 still took about 20 minutes (NVIDIA, 2026).

So the first practical question in agentic RL arrives before the loss function. We need to know how much it costs to produce a trajectory, whether it really came from the environment we intended to train on, and whether an environment failure might be misrecorded as a policy failure.

The Real Difficulty of Long Trajectories Is Credit Assignment

For a coding agent, the most natural reward is still the final outcome. A patch that passes the tests receives $1$, and any other patch receives $0$:

\[R(\tau)= \begin{cases} 1, & \text{the submitted patch passes the tests},\\ 0, & \text{otherwise}. \end{cases}\]

This outcome reward has an important advantage: it directly represents task success and is often harder to exploit through superficial behavior than a complicated process reward. The problem is that a trajectory may contain dozens of tool interactions and hundreds of thousands of tokens, yet end up represented by one scalar.

Suppose the agent behaves roughly as

\[\tau^{+} = [ \text{correct search}, \text{correct edit}, \text{unnecessary detour}, \text{recovery}, \text{success} ],\]

and receives

\[R(\tau^{+})=1.\]

There is clearly behavior in this successful trajectory that should not be encouraged. Conversely, the agent may produce

\[\tau^{-} = [ \text{correct hypothesis}, \text{correct search}, \text{correct edit}, \text{bad final validation} ],\]

and receive

\[R(\tau^{-})=0.\]

Most of the behavior in this trajectory is worth preserving.

This is why credit assignment is especially difficult for long-horizon agents. As we saw in the GRPO section, all sampled tokens in one response usually share the same response-level relative advantage:

\[\hat A_{i,t}=\hat A_i.\]

For a short reasoning response, this approximation can work very well. As the trajectory becomes longer, its internal behavior becomes more heterogeneous. Broadcasting one terminal outcome across the entire trajectory makes it increasingly likely that bad behavior in a positive sample and good behavior in a negative sample are reinforced or suppressed together.

These problems are often more visible in real trajectories than in the formula. The training curve may look normal, entropy may not have collapsed, KL may show no anomaly, and average reward may even rise slowly, while held-out agent performance remains unstable. Changing the clip range or learning rate may not reach the root cause. Once successful and failed trajectories are opened up, the first question is often: which part of the behavior is the current advantage actually rewarding?

DeepSWE recorded a concrete failure mode. An agent may have found a patch that passes the tests within its first ten steps, then continue making random file changes and eventually destroy the correct result. This phenomenon is related to termination and directly exposes the credit-assignment problem: the final outcome cannot automatically distinguish the useful prefix from the destructive suffix.

Similar issues appear in overlong trajectories. In mathematical reasoning, a response that exceeds the context limit may simply not have had time to emit its final answer, so DAPO-style overlong filtering or soft punishment can reduce noisy reward from truncation. For a coding agent, an overlong trajectory can mean something else: the model has entered a loop of repeated searches, repeated edits, and repeated attempts to fix the same failed test. If all overlong samples are filtered, they may happen to be a particularly valuable set of negative behaviors.

There is therefore no “correct length” independent of trajectory semantics. Training needs to know why an episode ended: an explicit Finish, test success, context exhaustion, timeout, maximum steps, or environment failure. All may produce a $0$, yet they do not represent the same policy behavior.

It is also unnecessary to solve perfect per-token causal attribution immediately. A useful practical compromise is to find a sufficiently trustworthy local boundary. Suppose a trajectory contains an obvious first error:

\[[ \underbrace{a_0,\ldots,a_k}_{\text{useful prefix}}, \underbrace{a_{k+1},\ldots,a_T}_{\text{bad suffix}} ].\]

Even without proving the exact causal contribution of every action, it may already produce a cleaner learning signal if we avoid suppressing an obviously correct prefix because of the final failure. One option is to keep the prefix and roll out the suffix again from that state. A more general option is to learn a prefix-dependent value.

That brings the problem back to the critic we encountered earlier.

Once the Critic Returns, Credit Assignment Becomes Value Estimation

Looking back at PPO and GRPO, one of their key differences is the source of the advantage.

GRPO uses the relative performance of complete responses for the same prompt:

\[\hat A_i = R_i-\bar R,\]

and assigns this signal to the entire response.

PPO instead tries to estimate the future return from each prefix:

\[V^\pi(s_t) = \mathbb E_\pi[G_t\mid s_t],\]

and uses the TD error and GAE to obtain an advantage that changes with the state:

\[\delta_t = r_t+\gamma V_\phi(s_{t+1})-V_\phi(s_t).\]

The value model is therefore not a perfect answer to causal attribution. It cannot prove that “the edit at step 17 caused the final success.” It does something different: as the trajectory unfolds, it estimates the expected future return for different prefixes, so different positions can at least receive different baselines.

If the expected return after an action changes from

\[V_\phi(s_t)=0.3\]

to

\[V_\phi(s_{t+1})=0.7,\]

then this transition provides more local information than the statement that the entire trajectory eventually succeeded. If there is no immediate reward and $\gamma=1$, the corresponding TD error is approximately $0.7-0.3=0.4$. Value estimation can therefore be viewed as a form of learned credit assignment.

This gives the earlier SAO design another practical interpretation. After SAO chooses single rollout, GRPO’s group baseline is no longer available, so the critic returns. At the systems level, this removes group synchronization. At the level of the learning signal, it also makes the advantage prefix-dependent again.

The central difficulty of single-rollout agentic RL therefore becomes: can we train a value model that is good enough?

Long-horizon agents are unfriendly to critics. The final reward may arrive only after dozens of tool interactions, with many policy decisions and environment observations between an early state and the terminal outcome. The actor is also changing continuously, so the target of

\[V^{\pi_{\theta_k}}(s)\]

changes with every policy update. If the critic learns too slowly, GAE uses an outdated baseline. If critic training is unstable, the advantages received by the policy become noisy directly.

This is why, in a single-rollout setting, the value model can no longer be treated as an optional side component of PPO. Value pretraining, more frequent critic updates, and restricting the critic’s own update range all address the same goal: make the learned prefix value more informative than a crude trajectory-level baseline and keep it synchronized with the actor.

The multi-turn environment adds another semantic layer to value estimation. The trajectory is

\[[a_0,o_1,a_1,o_2,\ldots],\]

where $a_i$ is a policy action and $o_i$ is an environment observation. The observation can substantially change the probability of future success, yet it was not sampled as a policy action. SAO therefore also needs to handle advantage propagation between actions and observations explicitly.

Looking back at the whole PPO to GRPO to SAO path, the disappearance and return of the critic reflect a change in the training setting. In response-level reasoning RL, GRPO replaces expensive value estimation with group comparison. Long-horizon agents make group synchronization expensive and trajectory-level credit coarse, which makes prefix-dependent value attractive again.

Does the Learner See the Same Trajectory That Rollout Produced?

Even with good credit assignment, there is a separate requirement: the $(s_t,a_t)$ used by training must really be the $(s_t,a_t)$ that occurred during generation.

This is easy to satisfy for a single-turn response. The model generates

\[z_1,z_2,\ldots,z_T,\]

in token space, and the training side saves these token IDs and their corresponding log probabilities.

In a multi-turn agent framework, model calls usually sit inside a message protocol. A generation may first be decoded into text and parsed into a structured tool call. After the tool executes, the observation is written back into the conversation, and the entire history passes through a chat template and tokenizer to become the input to the next model call.

Even when the tokenizer never changes, there is no guarantee that

\[\operatorname{encode} \left( \operatorname{decode}(z_{1:k}) \right) = z_{1:k}.\]

NVIDIA’s NeMo Gym documentation records exactly this kind of problem: a set of token IDs actually generated in the previous round may become a different set of IDs after decoding to a string and tokenizing again in the next round. Re-rendering a structured tool-call object through the chat template can also change the formatting of the original generation (NVIDIA, 2026).

Suppose rollout actually produced

\[s_t^{\mathrm{rollout}} \rightarrow a_t,\]

while the training side reconstructs another prefix from the final conversation text:

\[s_t^{\mathrm{train}} \neq s_t^{\mathrm{rollout}}.\]

Then even if the model weights are identical,

\[\pi(a_t\mid s_t^{\mathrm{train}})\]

and the probability recorded during rollout,

\[\pi(a_t\mid s_t^{\mathrm{rollout}})\]

are no longer the same conditional probability.

This comes from state reconstruction in the rollout pipeline and is unrelated to asynchronous policy lag. It shows that text is not a lossless representation of an agent trajectory.

The solution is also engineering-oriented: treat rollout-time data as the source of truth. For every model call, save the actual prompt_token_ids, the sampled generation_token_ids, and the corresponding generation_log_probs. The training side should not infer what happened from the final conversation text.

Context summarization or compaction makes the problem more severe. Retokenization changes the token representation of the same semantic history. A summary directly changes the state seen by the next model call. A long-horizon trajectory is therefore not always one monotonically growing token sequence:

\[z_1,z_2,\ldots,z_T.\]

It is more accurately a sequence of real model calls:

\[s_0\rightarrow a_0, \qquad s_1\rightarrow a_1, \qquad \ldots\]

Each $s_t$ may be reconstructed after a tool output, truncation, or compaction. Training needs to save the states that actually occurred.

The More Expensive the Rollout, the More Valuable Freshness Becomes

With a faithful trajectory in hand, we still need to ask when it enters the learner.

The time boundary in standard synchronous PPO is very clean. First freeze the rollout policy, collect a complete batch, and save the sampled actions and rollout-time probabilities. Only after all rollouts finish does the optimizer begin. The main benefit is that the source of the evidence is clear and the data is relatively fresh.

The problem with agentic rollouts is large latency variance. A simple task may finish in a few minutes, while another trajectory may search, compile, test, and recover for twenty minutes. As long as the training loop retains a synchronous barrier, completed trajectories, rollout workers, and the learner all have to wait for the slowest stragglers.

This is why asynchronous rollout is attractive. Let completed trajectories enter the learner quickly and overlap generation with optimization to improve system utilization.

The cost is freshness.

Suppose a trajectory starts under

\[\pi_1,\]

and by the time it reaches the learner, the current policy has become

\[\pi_4.\]

Even if the trajectory used $\pi_1$ from start to finish, it is still stale evidence. The earlier SAO section showed a more aggressive case: if the rollout engine synchronizes new weights before a trajectory is complete, that same trajectory may span multiple policy versions.

Importance correction can handle part of the distribution mismatch. It cannot restore the information quality of a fresh rollout to arbitrarily stale evidence. As policy drift grows, importance ratios move farther from $1$, more samples enter clipping or masking regions, and the effective signal that an expensive trajectory contributes to the optimizer may shrink.

In long-horizon agentic RL, the degree of on-policy training can therefore be viewed as a systems budget.

Maintaining freshness more strictly requires more synchronization and more worker idle time. Overlapping rollout and training more aggressively improves throughput, while accepting greater policy lag and relying more heavily on correction, masking, and sample discard.

NVIDIA’s preliminary observation on SWE RL captures this tension well. They initially wanted to allow more off-policy steps to improve system utilization, but early experiments found that four off-policy steps were already too many for SWE RL, so they temporarily used a more conservative configuration. The number is not a universal threshold. It illustrates a more basic issue: SWE rollouts are expensive, so reusing them is highly attractive, while the policy changes quickly enough that the useful lifetime of this expensive evidence is shorter than expected.

This creates a practical trade-off. Filtering stale trajectories can improve data quality while reducing effective sample throughput. Replacing the discarded data requires more rollouts, which are precisely one of the most expensive parts of the system. At this point, data filtering, policy freshness, and infrastructure utilization begin to affect one another.

After Reward Rises, Look at Behavior First

Even when the environment, credit assignment, trajectory fidelity, and freshness are under control, one commonly underestimated question remains: what does a rising training curve actually mean?

The agent may truly have learned a better search or debugging strategy. It may also have found a shortcut in the verifier. It might modify tests without fixing the implementation, exploit an abnormal exit path to bypass a checker, repeatedly call tools that produce an easy shaping reward, or learn a fixed procedure that works only inside the training harness.

I therefore treat the reward curve as an observation that needs interpretation, together with capability metrics that indicate whether the model has actually improved.

This is especially important for long-horizon agents. A longer trajectory does not automatically mean stronger reasoning. More tool calls do not automatically mean better tool use. More reflection-like language, more search, and more retries all require a further question:

\[P(\text{success}\mid\text{this behavior})\]

has this probability actually increased?

Reward design and trajectory auditing are therefore difficult to separate. Before training, inspect a sample of positive and negative trajectories by hand to verify that the verifier’s labels match our understanding of task success. During training, keep sampling failed trajectories to see where the agent is actually failing.

RAGEN-2 gives a subtler example. An agent’s token entropy can remain high while the model gradually produces fixed reasoning templates unrelated to the input. They call this phenomenon template collapse and supplement entropy diagnosis with a mutual-information-style cross-input distinguishability measure. Output diversity for one input does not guarantee that the model still changes its strategy in response to different inputs (Wang et al., 2026).

A similar phenomenon is easy to imagine for a coding agent. Its shell commands may look different across repositories, yet it may repeat the same policy:

\[\text{grep} \rightarrow \text{read README} \rightarrow \text{run tests} \rightarrow \text{edit first match} \rightarrow \text{run tests}.\]

Token-level diversity still exists, while the policy’s real sensitivity to the environment state has declined.

Useful observability during training therefore comes from behavior. Record success rate, trajectory length, tool distribution, invalid calls, repeated actions, termination reasons, and reward variance. Metrics are only the entry point. The more important task is to align them with actual task success and periodically inspect real trajectories to see what the policy has changed.

Many agent-RL failures cannot be explained by a scalar dashboard alone.

In Practice, the Process Looks More Like a Debugging Loop

Putting these issues together, I would hesitate to summarize long-horizon RL as one fixed recipe. Real training looks more like repeatedly debugging a closed loop:

\[\text{rollout} \rightarrow \text{inspect} \rightarrow \text{find failure mode} \rightarrow \text{change the loop} \rightarrow \text{rollout again}.\]

The first step is usually to fix the environment and verifier. Run a baseline agent on enough trajectories to confirm that the tool parser, sandbox, tests, timeouts, and reward do not contain an obvious shortcut or random failure. Otherwise, environment noise can flow straight into the policy gradient.

Next, confirm that the model has minimum viable agent behavior. If tool syntax, search, editing, and the test loop are not reliable, RL will produce almost no meaningful positive evidence. This stage is better served by SFT, skill training, or shorter-horizon tasks that establish a cold start.

The training distribution can then expand gradually. Start with tasks on which the policy occasionally succeeds, then increase the horizon, tool complexity, and context budget. Long trajectories can themselves serve as a curriculum dimension because the more state the model must maintain, the more difficult it becomes to assign terminal reward to early decisions.

If the reward curve does not rise, inspect the trajectories first. The problem may be insufficient exploration, or it may simply be that many all-failure rollouts provide no signal. If reward rises while evaluation does not, examine the verifier, credit assignment, and behavioral collapse. If many correct prefixes are penalized together with a final error, the problem may be credit. If the value model cannot keep up with the actor, the problem may be the critic. Only when trajectories are expensive and the learner is continually waiting for workers does it make sense to increase asynchronous overlap further.

When shaping is genuinely needed, denser signals can be added. Every intermediate reward should come with the question: has the policy started optimizing this proxy? Any change to termination, filtering, sampling, context compaction, or environment logic should also be treated as a change to the training distribution, rather than merely an infrastructure refactor.

Only then should asynchronous throughput be pushed more aggressively. At that point, monitor rollout latency, policy freshness, importance ratios, environment failure rate, critic quality, held-out success, and real failure trajectories together. A faster system is useful only if its trajectories still constitute useful evidence in the end.

From this perspective, many practical questions in agentic RL fall into three distinct levels.

Credit assignment asks:

\[R(\tau) \quad\text{should become}\quad A_t.\]

Trajectory fidelity asks whether the

\[(s_t,a_t)\]

used by the learner is really the state-action pair that occurred during rollout.

Policy freshness asks how far

\[\pi_{\mathrm{behavior}}\]

has moved from the current policy by the time the evidence reaches the optimizer.

The environment, rollout infrastructure, and evaluation jointly determine what the optimizer ultimately learns from these three questions.

This is the most visible change to me when moving from ordinary LLM RL to agentic RL. The policy-gradient formula from earlier still holds:

\[\nabla_\theta J = \mathbb E \left[ A_t \nabla_\theta \log\pi_\theta(a_t\mid s_t) \right].\]

The difficult part gradually moves to the other objects in the formula. Is this $s_t$ really the state the model saw? Which rollout policy produced $a_t$? And most importantly, did $\hat A_t$ assign the final outcome to the decisions that are actually worth learning from?

The policy objective remains the core of the training loop. In long-horizon agentic RL, though, the process that produces, interprets, stores, filters, and finally sends a trajectory to the optimizer jointly defines what the policy update means.

Summary: From Policy Optimization to the Agent Learning Stack

Looking back at PPO, GRPO, and the variants that followed, I increasingly feel that PPO already provides a fairly complete RL solution. It uses a value model to estimate the expected return of different prefixes, GAE to construct token-level advantages, a probability ratio to handle the mismatch between rollout and current policies, and clipping to limit how far one batch of fixed evidence can push the policy. The problem is that the solution is heavy. The critic requires extra model capacity and training cost. Value targets contain sampling noise. The actor, critic, and rollout policy need to maintain strict data relationships. Once this is placed on top of an LLM, memory, throughput, and training stability become practical constraints.

Many methods that appeared after reasoning RL can be understood as cheaper approximations under these constraints. GRPO is the clearest example. If several responses can be sampled for the same prompt at reasonable cost and the final verifier is reliable enough, the samples in the group can provide a baseline directly:

\[\hat A_i = R_i-\bar R.\]

There is then no need to train an additional

\[V_\phi(s_t).\]

Dr. GRPO, DAPO, GSPO, CISPO, and SAPO continue modifying normalization, sampling, loss aggregation, importance ratios, and clipping. Each addresses a different problem, yet together they show that in a reasoning-RL setting, some of PPO’s expensive components can be replaced by simpler estimators or training recipes, often with a favorable trade-off.

The balance starts to change again in long-horizon agentic RL.

A rollout now contains a complete interaction beyond the response:

\[\tau = (s_0,a_0,o_1,s_1,a_1,o_2,\ldots,s_T),\]

where dozens or hundreds of model actions and environment observations jointly determine the final outcome. A successful trajectory may contain many bad decisions, while a failed trajectory may still contain a long correct prefix. If the entire trajectory continues to share one group-level advantage, the reward becomes increasingly weak at explaining local behavior.

This is why credit assignment becomes prominent again in agentic RL. The problem is no longer only obtaining

\[R(\tau)\]

at the end of the trajectory. We also need to turn that terminal outcome into learning signals for different decisions inside the trajectory:

\[R(\tau) \longrightarrow \hat A_t.\]

Perfect causal attribution is not always necessary. If we can avoid penalizing an obviously correct prefix because of a later failure, or avoid reinforcing every clearly ineffective action after a final success, partial credit may already contain more information than one scalar shared by the whole trajectory. Value models, subtrajectory returns, process signals, and prefix replay all try to recover some of the trajectory structure compressed by a terminal reward.

SAO creates an interesting loop here. To avoid grouped-rollout synchronization, it switches to single rollout. When $G=1$, GRPO’s group baseline disappears naturally. Training then needs a learned critic, GAE, and behavior-policy correction again.

In a sense, we have returned to PPO.

What returns is PPO’s core structure: prefix-dependent value estimation, advantage estimation, and correction for old-policy evidence. The setting is now an agent trajectory produced by environment interaction, tool latency, context management, and multiple rollout-policy versions. SAO’s asynchronous rollout, per-token rollout probabilities, skip-observation GAE, and stale-sample masking are additional requirements imposed by this new setting on a PPO-like framework.

I therefore prefer to understand the changes across these algorithms as a movement of the bottleneck. PPO is complete, yet difficult to train. In reasoning RL, cheaper approximations may be good enough, so the cost of the critic and complex value estimation can be avoided temporarily. As agent trajectories become longer, group-level credit and synchronous sampling become new constraints, making the value model and more detailed trajectory bookkeeping worth their cost again.

From this perspective, what determines whether RL works is probably more than any one policy objective. Rather than looking for the next prettier objective, I find it more useful to view RL as a complete learning stack, as shown in Figure 4. Each layer answers a different question, and together they determine whether the learning signal received by the optimizer is worth trusting.

The stack can be understood layer by layer, following how a trajectory is produced and used.

1. Training data determines which problems we train on. RL does not automatically decide which tasks should be used for training. Given

   \[x\sim\mathcal D_{\mathrm{train}},\]

   task difficulty, domain coverage, horizon, tool complexity, and whether the current policy can produce enough successes and failures on these tasks directly determine how much useful evidence later rollouts can provide. Tasks that are too easy quickly become all-success, while tasks that are too difficult remain all-failure. Either case can leave the estimator without a useful direction. The data distribution determines the learning opportunities that the policy encounters during training.

2. The harness and environment determine how these experiences happen. Once a task is given, the model still needs to act within a concrete interaction process. Tool schemas, observations, message history, error handling, context management, termination logic, and environment transitions all change the state the model actually sees and what happens after an action. In agentic RL, these components directly participate in defining the state-action process faced by the policy.

3. The rollout distribution determines which trajectories are ultimately produced. Even with identical input tasks and environments, sampling temperature, context budget, parallelism, termination, tool latency, and the current policy still change

   \[p(\tau\mid x).\]

   Input data determines where training starts. The rollout distribution determines whether those problems become successful, failed, short, long, or anomalous trajectories. Together they form the data actually used by RL.

4. The reward and verifier determine which trajectories count as good. After the rollout finishes, the verifier compresses a complex interaction into a reward:

   \[\tau\longrightarrow R(\tau).\]

   The most important requirement at this layer is that the reward corresponds to the outcome we actually care about. If the verifier contains a shortcut, or an environment failure is recorded as a model failure, the optimizer has no extra information with which to reconstruct our intended objective. It will simply become more effective at increasing the reward it actually receives.

5. Credit assignment determines which decisions receive the outcome. Even with a correct reward, we still need to perform

   \[R(\tau) \longrightarrow \hat A_t.\]

   For a short reasoning response, a response-level signal may be enough. For a long-horizon agent, one terminal outcome often covers actions with very different quality. Bad behavior in a successful trajectory should not all receive positive credit, and a useful prefix in a failed trajectory should not all receive negative credit. As the horizon grows, this layer increasingly needs partial credit, subtrajectory estimates, or prefix-dependent value. That is why the value model becomes important again.

6. Trajectory evidence determines whether the learner receives the evidence that actually occurred. After the advantage is constructed, $s_t$, $a_t$, the action mask, and the rollout-time log probability still need to correspond to the interaction that actually happened during generation. If a multi-turn pipeline reconstructs another state during decoding, tool parsing, re-templating, or context compaction, the training-time probability no longer corresponds to the rollout-time conditional probability. The trajectory also cannot be so stale by the time it reaches the learner that it is nearly unrelated to the current policy. Fidelity and freshness jointly determine whether rollout evidence is still statistically worth using for an update.

7. Policy optimization determines how the signal updates the policy. Only now do we arrive at PPO, GRPO, SAO, and other policy optimizers. They receive the already constructed $\hat A_t$ and use

   \[\hat A_t \nabla_\theta \log\pi_\theta(a_t\mid s_t)\]

   to change the probabilities of sampled actions. Ratios, clipping, masking, and weighting rules determine how existing evidence is used and how far one update can trust it.

If the entire stack had to be compressed into one line, I would write

\[\text{good data} \rightarrow \text{good interaction} \rightarrow \text{good reward} \rightarrow \text{good credit} \rightarrow \text{good evidence} \rightarrow \text{good optimization}.\]

It is difficult to say that one layer is always the most important. Phrases such as “data is all you need,” “credit assignment is all you need,” or “infra is all you need” usually describe the dominant bottleneck at a particular stage. Once the other components are good enough, the weakest layer naturally determines whether the training curve can continue to improve.

If we look at some current agentic RL training practice, credit assignment seems especially worth checking. A recurring contrast is that reasoning RL often does not need elaborate credit assignment. As long as the task distribution is difficult enough, group sampling produces useful reward variation, train-inference consistency and exploration are sound, and the GRPO-style response-level signal can often support stable policy improvement. The same recipe can become much harder in long-horizon agentic tasks. Some public practice reports describe training statistics that look largely normal, with no obvious issue in the rollout system, entropy, KL, or train-inference consistency, while held-out evaluation still fails to improve reliably. Further changes to data, batch size, rollout count, or common regularization do not necessarily solve the problem directly.

Putting this observation back into the RL stack, I would first inspect credit assignment. GRPO copies one response-level advantage to every action token in the response:

\[\hat A_{i,t} = \hat A_i.\]

For an agent trajectory with dozens or hundreds of interaction rounds, this approximation places decisions with very different properties under the same update direction. If the task succeeds, clearly wrong, redundant, or harmful behavior in the middle is reinforced along with the positive advantage. If the task fails, reasoning, search, and tool calls that were correct earlier receive the same negative advantage. The reward may have judged the final outcome correctly. The information is mainly compressed in the step

\[R(\tau) \longrightarrow \hat A_t.\]

Clipping, importance ratios, and optimizer recipes determine how an existing advantage enters the update, yet they cannot recover local distinctions that have already been mixed into one scalar.

A recent and intuitive direction is to use positions that are relatively easy to identify inside the trajectory to split a long trajectory into more meaningful learning units. A first error, a clear pivot, or a high-confidence change in outcome can serve as a cut point. PivoARL (Guo et al., 2026), PivotRL (Yi et al., 2026), and TreeRL (Hou et al., 2025) all explore this direction, with different pivot signals and search or retry mechanisms. Their shared intuition is that once we know behavior begins to diverge from the goal after a particular prefix, the entire trajectory does not need to keep sharing the original outcome signal. The prefix can be fixed or reused, while the later part is rolled out, evaluated, and optimized again.

Prefix replay (Liao et al., 2026) also appears in a neighboring multi-turn post-training setting. The core operation is similar: keep an already generated prefix as the context for later model calls and concentrate the training budget on the suffix that still needs improvement. This can reduce repeated environment interaction and focus the learning signal on the unresolved part.

The other route is closer to PPO’s original answer: learn a prefix-dependent value again.

\[V_\phi(s_t) \approx \mathbb E_\pi[G_t\mid s_t],\]

and use GAE to obtain an $\hat A_t$ that changes with the state. This is also the direction taken by SAO after losing GRPO’s group baseline. Single rollout lets each trajectory enter the learner sooner, while making credit assignment substantially a value-estimation problem again.

Bringing the critic back does not solve everything automatically. The critic must learn a meaningful prefix value from sparse, long-horizon returns and keep up with the changing actor. The value model’s cold start therefore deserves attention. For example, existing offline rollouts or filtered training data can be used for value pretraining before online RL. Small-scale practice also suggests that critic quality and final actor performance do not always correspond simply. Value pretraining, data selection, and update frequency can all change the relationship. The same filtered data can help the actor start from a more suitable task distribution and give the critic easier value targets. Some SAO-related practice reflects this connection as well.

Learned value also needs its own diagnostics. A natural question is whether the critic explains return variation better than a simple baseline. If a learned $V_\phi(s_t)$ provides no more information than a prompt-level or group-level mean, the finer-grained advantage it produces may not lead to better credit. EVPO (Pan et al., 2026) further uses explained variance as a signal of critic reliability, helping decide when to rely more on the learned value and when to retain a simpler baseline.

Taken together, these methods suggest two broad routes for credit assignment in long-horizon agentic RL. One route uses pivots, cut points, or prefix replay to isolate the most obvious credit errors from the whole trajectory. The other trains a prefix-dependent value so that the critic supplies a different expected-return estimate for each state. The first is easier to use as a low-cost local correction. The second takes on the training cost and stability problems of a value model. The choice depends on trajectory length, available intermediate signals, and whether the system is willing to maintain an additional critic-training pipeline. It is therefore natural for SAO to return to PPO-like training. Components that look expensive in PPO begin solving real problems again in this task regime.

Of course, this is only one example of the bottleneck moving through the stack. Algorithms and infrastructure are both essential. Without a stable rollout system, enough environment throughput, and accurate trajectory bookkeeping, long-horizon agent experiments cannot run, and none of the data, reward, or credit choices above can matter. The layer that limits training will continue to change across tasks, model scales, and resource conditions.

There is one more element that does not appear as a vertical layer in this stack but runs through almost every stage: observability. This is why Figure 4 draws it as a side rail rather than as one of the layers. It reads the entire learning loop without directly producing the training signal for the next layer.

After reward rises, we need to know which behavior the agent actually changed. When evaluation does not rise, we need to distinguish input distribution, environment noise, reward shortcuts, bad credit, critic error, and stale rollout. Success rate, trajectory length, tool-call distribution, termination reason, invalid actions, and held-out evaluation provide different kinds of evidence. We still need to return to real trajectories to see what the policy has learned. Training curves that look identical can correspond to completely different failure modes.

As agentic RL progresses, another question becomes increasingly important. We often write the policy as

\[\pi_\theta(a_t\mid s_t),\]

but a real agent’s $s_t$ is heavily shaped by the harness. The system prompt, tool schema, message protocol, memory, context compaction, error handling, and termination logic all change the state that the model actually sees at each step.

The object being trained therefore increasingly resembles

\[\text{Agent} = \text{Model} + \text{Harness} + \text{Environment}.\]

This raises an open question. If RL always happens under a fixed harness, how much of what the policy learns is a transferable agent capability, and how much is bound to one tool interface, context organization, or interaction protocol? Should future agentic post-training happen across multiple harnesses and environments, or should it specialize more strongly to the deployment harness? There is no unified answer yet.

One point is becoming clearer. After writing the whole article, return to the policy gradient from the beginning:

\[\nabla_\theta J(\theta) = \mathbb E \left[ \hat A_t \nabla_\theta \log\pi_\theta(a_t\mid s_t) \right].\]

The formula itself has not become much more complicated. The difficult part is making every object in it trustworthy. Those are the objects indicated by the brackets on the right side of Figure 4: does the training distribution provide suitable problems and experiences? Did the trajectory occur in the right interaction process? Does the reward really represent success? Did $\hat A_t$ assign the outcome to the right decisions? And when the state-action evidence reaches the learner, is it still real and useful?

That may be the biggest change from reasoning RL to agentic RL. We started by studying how to write a better policy update. As trajectories, environments, and harnesses become more complex, RL gradually becomes the design of an entire agent-learning system.