Summing agent rewards before normalizing lets the noisiest channel drown out guardrails

The trouble starts when GRPO has more than one reward to work with. Vanilla GRPO adds the channels together first and then normalizes the total. The authors call this step Sum-then-Normalize ^[8]. Most of the spread in that total comes from whichever channel varies the most. The post calls the result scale dominance: a high-variance channel drowns out subtle constraint metrics during joint normalization ^[5]. Suppose task success swings hard across a group while a JSON-schema score barely moves. Schema compliance then shifts each rollout's advantage very little.

The second failure happens even with a single channel. When all 8 rollouts in a group fail, every reward is 0 and the group standard deviation is 0, so every advantage in the group is 0 ^[9]. Those rollouts cost full price to generate and produce no gradient.

PPO does not use group statistics. It trains a critic to predict expected future reward ^[10]. According to the post, a 27B policy usually comes with a 27B critic, with both models and their optimizer states and activations held in GPU memory, which takes twice as many H100s ^[11]. Together the two models hold 54B parameters before any optimizer state is counted ^[15]. Nobody enjoys renting H100s for a network whose only job is guessing the score. The authors also say a single critic head predicting a blend of accuracy, latency penalties and format compliance is unstable. In their account, advantages get noisy and policy updates slow down ^[12]. GRPO dropped the critic and cut memory needs in half ^[7].

The authors propose two changes. The first is GDPO, which decouples normalization by channel ^[13]. Each channel is scaled against its own spread within the group before the channels are combined. A guardrail score that varies only a little can then still move the advantage when rollouts differ on it.

The second change is that hard constraints must be enforced and not carried as reward weights ^[14]. I agree with that choice for guardrails. A weighted penalty is a number that a large enough task reward can outweigh, and the post warns that agents learn to game summed rewards ^[4].

The title lists four trainers ^[16]. The authors' Qwen3-14B run is separate from a 27B results table ^[2]. The available text of the post breaks off before it describes DAPO or reaches either table. The authors say the Qwen3-14B run does not establish a universal trainer ranking ^[2]. A ranking from that run carries over to another team only if that team's reward channels have a similar spread and its tasks produce all-fail groups at a similar rate. Those are the two conditions behind the failures the post describes ^[5][9].