Two Things RL Beginners Get Confused About

Over the past few months I have had a lot of technical discussions with colleagues who are ramping up on RL for LLM post-training. Most of them come from a supervised-learning background, and two comments from those conversations stuck with me, because each one is correct about the math on the page and still leads to the wrong picture of what RL does. I ended up digging into both much more deeply than I expected. Since they are probably common among people new to RL, I am writing them down here.

Comment 1: “RL is just weighted SFT.” This came from a colleague who had just worked through a policy-gradient derivation. The SFT loss and the REINFORCE loss, as you would write them in code, look like this:

\[\mathcal{L}_{\mathrm{SFT}}=-\frac{1}{N}\sum_{i=1}^{N}\log\pi_\theta(y_i),\qquad y_i\sim\text{dataset};\] \[\mathcal{L}_{\mathrm{RL}}=-\frac{1}{N}\sum_{i=1}^{N}R(y_i)\,\log\pi_\theta(y_i),\qquad y_i\sim\pi_\theta .\]

His reading: “The only difference is the \(R(y_i)\) in front. You sample from the model, pretend the samples are labels, and weight each one by its reward. It’s SFT with a multiplier.” He was right about the math, and that is exactly what makes the comment hard to answer.

Comment 2: “The loss function means nothing.” This came from a colleague reading OpenAI’s Spinning Up, where the same \(\mathcal{L}_{\mathrm{RL}}\) is introduced together with a warning box¹ that says, among other things:

“Even though we describe this as a loss function, it is not a loss function in the typical sense from supervised learning. […] It doesn’t measure performance. […] After that first step of gradient descent, there is no more connection to performance. This means that minimizing this ‘loss’ function, for a given batch of data, has no guarantee whatsoever of improving expected return. You can send this loss to \(-\infty\) and policy performance could crater; in fact, it usually will. […] The loss function means nothing.”

She found this confusing rather than clarifying, and I think her questions were the right ones:

If the loss means nothing, why do we minimize it?
If it has the right gradient, what exactly goes wrong after the first step?
And how far is too far?

Put the two comments side by side and there is already something odd. The first says \(\mathcal{L}_{\mathrm{RL}}\) is just SFT with a weight, so surely it behaves like a loss. The second says it is not a loss at all. Both are about the same two lines of code. They look unrelated, but they turn out to be two sides of the same fact, and once you see that fact both of them resolve at once. This post works through them in a world small enough that everything can be computed by hand.

A world with three answers

There is one prompt, and the model can answer it in only three ways:

answer	what it is	reward \(R\)	base model \(\pi_\base\)	SFT demos \(p\)
A	the best answer	\(1.0\)	\(0.1\)	\(0.5\)
B	an acceptable answer	\(0.6\)	\(0.6\)	\(0.5\)
C	a wrong answer	\(0.0\)	\(0.3\)	\(0\)

A policy is a distribution \(\pi=(\pi_A,\pi_B,\pi_C)\), produced by a softmax over three logits, which are the parameters we train. The base model is \(\pi_\base=(0.1,\,0.6,\,0.3)\): it usually says B, sometimes C, and rarely produces the best answer. This is the realistic case: if the base model already gave the best answer most of the time, there would be little left to train.

SFT is given demonstrations, say \(p=(0.5,0.5,0)\), and maximizes their log-likelihood. RL is given the reward table and maximizes the expected reward of the model’s own answers:

\[J_{\mathrm{SFT}}=\sum_k p_k\log\pi_k=\E_{a\sim p}[\log\pi(a)],\] \[J_{\mathrm{RL}}=\sum_k \pi_k R_k=\E_{a\sim\pi}[R(a)] .\]

Keep an eye on what sits under each expectation: in SFT a fixed \(p\), in RL the very \(\pi\) we are training. Everything in this post follows from that one difference.

Part 1: Comment 1, “RL is just weighted SFT”

The logic of this part is short enough to state up front. Per sample, Comment 1 is right. But a loss is an average over samples, and the two losses average over different things: SFT over a fixed dataset, RL over the model’s own outputs. Once you take that average, the two gradients have different forms, and the different forms produce different behavior. The rest of this part fills in each step.

Step 1: per sample, the comment is right

Take one sampled answer \(y\) and differentiate its term in each loss:

\[\nabla_\theta\big[-\log\pi_\theta(y)\big]=-\nabla_\theta\log\pi_\theta(y),\] \[\nabla_\theta\big[-R(y)\log\pi_\theta(y)\big]=-R(y)\,\nabla_\theta\log\pi_\theta(y).\]

Same vector, scaled by \(R(y)\). The RL update pushes up the log-probability of the sampled answer exactly as SFT pushes up the log-probability of a label, only harder or softer depending on the reward. At the level of a single sample, Comment 1 is simply correct.

Step 2: a loss is an average, and the averages run over different things

But nobody trains on one sample. The loss is an average over a batch, in the limit an expectation, so the gradient that actually moves the parameters is

\[\nabla J_{\mathrm{SFT}}=\E_{y\sim p}\big[\nabla\log\pi_\theta(y)\big],\] \[\nabla J_{\mathrm{RL}}=\E_{y\sim\pi_\theta}\big[R(y)\,\nabla\log\pi_\theta(y)\big].\]

In SFT the samples come from the dataset \(p\), which does not care what \(\theta\) is. In RL they come from \(\pi_\theta\) itself, so how often each answer appears in the average depends on the model being trained. Per sample this difference is invisible; in expectation it decides everything, and the three-answer world is small enough to see how.

The softmax gives \(\partial\log\pi_i/\partial\theta_k=\delta_{ik}-\pi_k\), where \(\delta_{ik}\) is \(1\) if \(i=k\) and \(0\) otherwise: raising one logit raises its own probability and lowers everyone else’s in proportion to their current size. Averaging over \(y\sim p\) for SFT and over \(y\sim\pi\) for RL,

\[\frac{\partial J_{\mathrm{SFT}}}{\partial\theta_k}=\sum_i p_i(\delta_{ik}-\pi_k)=p_k-\pi_k,\] \[\frac{\partial J_{\mathrm{RL}}}{\partial\theta_k}=\sum_i \pi_i R_i(\delta_{ik}-\pi_k)=\pi_k\,(R_k-\bar R),\]

with \(\bar R=\sum_i\pi_iR_i\) the current expected reward. Writing \(\dot\theta_k\) for the rate at which logit \(k\) moves under gradient ascent with a small step, the two update rules are

\[\text{SFT:}\quad \dot\theta_k=p_k-\pi_k\] \[\text{RL:}\quad \dot\theta_k=\pi_k\,(R_k-\bar R).\] (1)

Look at where the \(\pi_k\) in the RL rule comes from. The per-sample update for answer \(k\) has no \(\pi_k\) in front of it; if anything, a rare answer gets a bigger push when it is drawn, because of the \((1-\pi_k)\) on its own logit. The \(\pi_k\) is how often answer \(k\) shows up in the batch, which is how likely the model already is to produce it. In SFT the corresponding factor is \(p_k\), fixed by the dataset. The \(\bar R\) was not subtracted by hand either; it comes out of the softmax normalizer, and it means an answer is pushed up or down according to whether it beats the model’s own current average, not its absolute reward.

Step 3: reading the two rules

The SFT rule says: move each logit toward its target, with a force equal to the gap between where the probability is and where it should be. An answer the model never produces but the demonstrations contain gets a full-size push. SFT can put mass where there is none.

The RL rule says: move each logit in proportion to two things, how much better than average the answer is (\(R_k-\bar R\)) and how likely the model already is to produce it (\(\pi_k\)). A rare answer barely moves, no matter how good it is; a common answer moves fast. RL can only redistribute mass among answers it already produces. In plain English: rich get richer. (Biologists will recognize the form of the replicator equation, \(\dot\pi_k=\pi_k(R_k-\bar R)\). That equation moves probabilities, though; ours moves logits, which turns out to be even more rich-get-richer.)

Where does each rule stop? SFT stops when \(\pi=p\): the gaps are all zero and the model has matched the demonstrations, A half the time and B half the time, even though A is strictly better. RL stops when \(\pi_k(R_k-\bar R)=0\) for every \(k\): every answer that still has probability must have reward equal to the average, which can only happen when all the probability sits on the best answer (or, if several tie for best, on those). You can see the same thing without calculus: \(J_{\mathrm{RL}}=\pi_A\cdot1.0+\pi_B\cdot0.6+\pi_C\cdot0\) is a weighted average of the rewards, and a weighted average is largest when all the weight sits on the largest term. From the base model, \(J=0.46\), move a tenth of the mass from C to A and get \(0.56\); nothing stops you until the corner \((1,0,0)\). SFT ends by matching a distribution; RL ends by committing to the best answer (Figure 1).

Two triangles of all policies over answers A, B and C. Left, the SFT objective: curved level sets with the maximum at the demonstration distribution p, on the edge between A and B. Right, the RL objective: straight parallel level sets with the maximum at corner A. — **Figure 1:** Both objectives drawn over the triangle of all policies (corners = deterministic policies). SFT climbs a curved hill whose top is \(p\); RL climbs a tilted plane whose top is a corner.

Step 4: the path, not just the endpoint

Figure 2 runs both rules from the base model, and the middle panel is the one to look at. RL does not head straight for A. It first drives C out and piles the mass onto B, the acceptable answer it was already producing; A barely moves because \(\pi_A\) is small. Only after C is gone does A start to grow, slowly, at a rate set by its own small probability. The rare good answer is exactly the one RL is slow to find. In the right panel, where A and B are equally correct, RL does not cover them; it concentrates on B, the one it already liked, because while C is pushed out the two good answers grow in proportion to their current size.

Three trajectories on the policy triangle with the probability of A, B and C over time below each. SFT settles at half A, half B. RL with reward (1, 0.6, 0) first rises toward B, then turns to A, ending near A. RL with binary reward (1, 1, 0) ends almost entirely on B. — **Figure 2:** Gradient ascent from the base model \((0.1,0.6,0.3)\) under the two rules in (1). **Left:** SFT settles at the demos. **Middle:** RL first sharpens onto B, the common acceptable answer, and only slowly discovers the rare best answer A. **Right:** with a binary reward where A and B are both correct, RL concentrates on B, the one it already preferred; A shrinks to 3%.

Part 2: Comment 2, “The loss function means nothing”

What the loss is, and is not

Here is the REINFORCE loss as written in code. Samples \(a_i\) are drawn from the current policy \(\pi_\old\), and each gets an advantage \(A_i=R(a_i)-\bar R_\old\): its reward minus the old policy’s average (no relation to answer A). Advantages average to zero under \(\pi_\old\). Subtracting the average leaves the expected gradient unchanged, which is why \(\bar R\) showed up in (1) on its own, but it makes each sample’s push less noisy. Then:

logp = log_prob(model, a_i)              # log pi_theta(a_i)
loss = -(A_i.detach() * logp).mean()     # gradient flows through logp only

As a function of \(\theta\) with the batch frozen, this loss is \(-\tilde g(\theta)\), where

\[\tilde g(\theta)=\E_{a\sim\pi_\old}\big[A(a)\log\pi_\theta(a)\big].\]

Two facts about this function are all you need, and together they are what the warning box in Comment 2 is saying:

Its gradient at \(\theta=\theta_\old\) is \(\E_{a\sim\pi_\old}[A(a)\nabla\log\pi_\old(a)]=\nabla J_{\mathrm{RL}}(\theta_\old)\), the true policy gradient. That is the reason the loss is written this way.
That is the only thing it does. The value of \(\tilde g\) is not the expected reward, nor an estimate of it. It is linear in \(\log\pi_\theta\) and so unbounded: push \(\log\pi_\theta(a)\to-\infty\) for any sample with negative advantage and the “loss” improves forever, while the true return is bounded by the largest reward.

So the answer to “why minimize something that means nothing” is that we don’t. We take its gradient at one point, where the gradient is right, and we never look at its value. It is a surrogate: a function built to have the correct slope at \(\theta_\old\) and nothing more. In SFT the loss and the objective are the same function. In RL they only share a slope, and only at one point.

What goes wrong after the first step

Suppose you ignore this and treat \(\tilde g\) as a loss anyway: collect a batch from \(\pi_\old=\pi_\base\) and keep taking gradient steps on it. To rule out sampling noise as the explanation, let the batch be infinitely large, so that everything below is an exact expectation.

Left: true return over 80 gradient steps. Resampling every step climbs to about 0.99; a fixed batch peaks near 0.63 and settles at 0.61; a fixed batch with PPO-style clipping levels off near 0.56. Right: for the fixed-batch run, the change in its loss grows in a straight line while the change in true return flattens after about twenty steps. — **Figure 3:** Optimizing on one batch drawn from the base model. **Left:** true return. Keep stepping on the same batch and the policy converges onto B (return 0.61); clip PPO-style and it moves a little, then freezes (once every answer’s probability has moved more than 20% in the direction its advantage points, every sample is clipped and the gradient is zero, which is the cue to draw a new batch); resample every step and it finds A. **Right:** the fixed-batch run seen two ways: its loss reports steady improvement forever while its true return stopped after twenty steps.

The fixed-batch run in Figure 3 is what the warning box describes. Its loss goes down at a constant rate, step after step. Its true return rises for twenty steps, to \(0.63\), then slides to \(0.61\) and stays: the model has converged onto B, the acceptable answer, and will never leave. The reason takes one line to see. The gradient of the frozen-batch loss with respect to the logits is

\[\frac{\partial\tilde g}{\partial\theta_k}=\sum_i\pi_{\old,i}A_i(\delta_{ik}-\pi_k)=\pi_{\old,k}A_k-\pi_k\underbrace{\textstyle\sum_i\pi_{\old,i}A_i}_{=0}=\pi_{\old,k}A_k ,\]

a constant vector that does not depend on \(\theta\) at all. Gradient steps on a fixed batch walk in a fixed direction forever, and the direction favors whichever answer had the largest \(\pi_{\old,k}A_k\) when the batch was drawn: \(0.6\times0.14=0.084\) for B, against \(0.1\times0.54=0.054\) for A. B wins because it was common then, and nothing in the batch can ever say otherwise.

Compare with the true rule from Part 1, \(\pi_k(R_k-\bar R)\): the same expression, but with \(\pi\) and \(\bar R\) updated as the policy moves. If you resample, \(\pi_A\) grows, so A’s push grows, and A eventually wins. If you do not, the sampling factor stays frozen at \(\pi_\old\). This is where Comment 1 and Comment 2 meet: the factor \(\pi_k\) that makes RL sharpen in Part 1 is the same factor that, left frozen, makes the loss lie in Part 2.

How far is too far: the TRPO bound

The warning box says the loss has “no connection to performance” after the first step. That is too strong, and my colleague was right to be bothered by it. There is a connection. It just weakens with distance, and the rate at which it weakens is known.

To state it, switch to the ratio form of the surrogate, which is what PPO (Proximal Policy Optimization) and TRPO (Trust Region Policy Optimization) use: \(g(\theta)=J(\theta_\old)+\E_{a\sim\pi_\old}[A(a)\,\pi_\theta(a)/\pi_\old(a)]\). It has the same gradient at \(\theta_\old\) as \(\tilde g\), but instead of a log it is an importance-sampling estimate: each old sample is reweighted by how much more or less likely the new policy is to produce it.

That reweighting already repairs the frozen sampling factor. In a one-step world like ours the ratio form is exact, \(g(\theta)=J(\theta)\) for every \(\theta\), so stepping again and again on the old batch with it finds A just as resampling does. What it cannot repair is a finite batch, where large ratios make the estimate noisy, and, as we will see below, a change in which states the policy visits. For those you need to know how far is too far. Kakade and Langford (2002) and Schulman et al. (2015) proved that for the true return \(f\),

\[f(\theta)\;\ge\;g(\theta)\;-\;C\cdot\dist(\pi_\old,\pi_\theta)\qquad\text{for every }\theta,\]

where \(\dist\) is a KL-type distance between what the two policies do and \(C\) is a constant that grows with the size of the advantages and the horizon.² At \(\theta_\old\) the distance is zero and \(g=f\). Away from it, the surrogate may overestimate the return, but by no more than \(C\) times how far the policy moved. Since \(g(\theta_\old)=f(\theta_\old)\), this becomes, for any new \(\theta_1\) however obtained,

\[f(\theta_1)-f(\theta_\old)\;\ge\;\underbrace{\big[g(\theta_1)-g(\theta_\old)\big]}_{\text{gain in the loss you can see}}\;-\;\underbrace{C\cdot\dist(\pi_\old,\pi_{\theta_1})}_{\text{price of leaving }\theta_\old}.\]

Both terms can be computed from the batch you already have. This is the bridge from the loss you can see to the return you cannot: the true return is guaranteed to improve whenever the gain in the surrogate beats the price of moving. TRPO maximizes \(g\) inside a KL ball so the price is capped; PPO clips the ratio, a cheap stand-in for the same cap. Plain REINFORCE takes one step of size \(\eta\) and checks nothing, and for it the bound says something simple: the gain is linear in \(\eta\), the distance is quadratic, so the right-hand side is positive for small steps and negative for large ones. For REINFORCE, the trust region is the learning rate.

To see the bound at work we need the smallest problem where the surrogate can be wrong even with infinite data: two decisions in a row, so that which states you visit depends on the policy.

A two-step decision problem. From the start state s0, action L (taken with probability 0.5) leads to state sL and action R (probability 0.5) leads to state sR. At sL, action u pays 0.52 and d pays 0.48, each taken with probability 0.5. At sR, action u pays 1.00 and d pays 0.00, and the old policy takes u with probability 0.35.

At \(s_0\) you go left (L) or right (R); then you pick up (u) or down (d), which pays a reward. Under the old policy, going left is worth \(0.5\) and going right only \(0.35\): u at \(s_R\) pays \(1\), but the old policy picks it just 35% of the time. So the policy gradient says two things at once: pick u more often at \(s_R\), and go left more often, because by the old policy’s standards left is the better branch.

Value against step size along the policy-gradient direction. The REINFORCE log-form loss rises in a straight line; the ratio-form surrogate rises and flattens near 0.82; the true return peaks at about 0.61 near step size 3 and then falls; the lower bound peaks at a small step near 0.6 and then drops steeply. — **Figure 4:** A two-step problem where the states the policy visits depend on the policy. Moving along the policy-gradient direction from \(\theta_\old\): the surrogate \(g\) keeps rising because it credits improvements at a state the old policy visited half the time; the true return \(f\) peaks and then falls as the new policy stops visiting that state at all. The green curve is the bound \(g-C\cdot\dist\); its maximizer is a step that is guaranteed to improve, and the best true step is invisible from old data.

Figure 4 moves along that gradient. The surrogate is a sum over states the old policy visited, and it cannot see that the new policy, by going left more often, has stopped visiting the state where the big reward lives. The bound is not a good estimate of \(f\); it sits far below it almost everywhere. What it gives you is a guarantee: the step it certifies is small and safe. The best true step is about five times larger, and nothing in the old data can tell you so.

Two caveats:

The distance is always measured to the policy that generated the batch, never to the previous iterate; this is why PPO’s ratio uses the rollout policy, held fixed across all the minibatch updates on a batch.
The bound is sufficient, not necessary: a large step can turn out fine, you just cannot tell from old data.

Part 3: The one fact behind both

In supervised learning the dataset is fixed and the model moves toward it. In RL the model generates the dataset, scores it, and moves; then the dataset is thrown away and a new one is generated for the next step. Comment 1 and Comment 2 are this loop seen from two different sides. (Spinning Up’s warning box actually says this first, “the data distribution depends on the parameters”; I left it out of the quote above because it is the punchline.)

Part 1 is what the loop does to the objective. The expectation is taken under \(\pi_\theta\), so differentiating it puts \(\pi_k\) (how often each answer is sampled) in front of every update, and that factor turns “match the good answers” into “amplify the good answers you already produce.” Sharpening, slow discovery of rare good answers, and the inability to reach answers the model never samples are all this one factor at work.

Part 2 is what the loop does to the loss. The batch was generated by one particular policy, so it is a faithful picture of the objective only at that policy. The frozen-batch gradient \(\pi_{\old,k}A_k\) shows what happens if you forget: the sampling factor stays glued to the policy that drew the batch, and the model walks off wherever the stale factor points. The trust region is the price of leaving the snapshot; resampling is how you take a new one.

	SFT	RL
Data come from	a fixed dataset	the current policy
Objective	concave; optimum \(=p\)	linear; optimum \(=\) a corner
Gradient on a logit	\(p_k-\pi_k\) (gap to target)	\(\pi_k(R_k-\bar R)\) (frequency \(\times\) advantage)
Batch loss estimates	the objective, at every \(\theta\)	the objective’s gradient, at one \(\theta\)
Epochs on one batch	fine; that is the goal	optimizes a stale snapshot
Read the loss curve?	yes	no; read the return
The model learns	to cover what the data do	to commit to what it already does well

A few practical habits follow from this:

Watch the return, not the loss.
Don’t run epochs on a rollout batch unless something (clipping) caps how far you move from the policy that produced it.
Expect the learning rate to be more delicate than in SFT, because for a plain policy-gradient step it is the trust region.
Expect entropy to fall and rare good behaviors to emerge slowly. That is the dynamics, not a bug, and it is why entropy bonuses, asymmetric clipping and sampling temperature are standard knobs.
Whenever you see “KL” in an RL paper, ask which one. The KL to a reference model in the objective is about the destination; the KL to the rollout policy in the trust region is about the step.