Dropping the critic network moves an RL fine-tune's cost from VRAM to rollout tokens

Start with the 600GB. The post puts that figure on a 70B actor plus a critic of roughly the same size, with AdamW optimizer states for both, and says the loop needs tensor and pipeline parallelism merely to fit ^[7]. Divide 600 by 70 and you get about 8.6 bytes per parameter to cover two trainable models and two sets of optimizer state ^[14]. The post says "over", so read it as a floor. It transfers to your cluster only if the critic really is actor-sized, the optimizer really is AdamW with full moment state, and nothing is being offloaded to host memory. Nobody deploys pipeline parallelism for fun.

The same post's case against scale is thinner: it asserts that high-signal public text is close to exhausted ^[2] and that going from 70B to 700B parameters spikes capital expenditure, interconnect overhead and power for marginal gains on everyday reasoning ^[3]. There is no measurement behind either claim in the text.

The infrastructure argument is the stronger half, and it is mostly about variance. A critic in this setting has to predict expected discounted return at every intermediate token of a trajectory that runs 8,000 to 16,000 tokens, and the post says critic error amplifies gradient variance until losses explode into NaNs ^[8]. GRPO, from DeepSeek's DeepSeekMath paper and scaled in DeepSeek-R1, replaces that estimate with the empirical distribution of G completions sampled for the same prompt ^[9]. Rewards come from the verifier: a regex answer parser or a compiler test run scores each completion ^[10].

That baseline has a failure mode the group size controls. If all G completions in a group earn the same scalar reward, each completion's advantage against the group mean is zero and the step carries no gradient signal ^[17]. Groups that never disagree are groups that never teach, so task difficulty and sampling temperature stop being tuning knobs and start being part of the training design.

Count the networks before assuming the saving is three quarters. PPO hosts four: trainable actor, trainable critic, frozen reference for per-token KL, frozen reward model ^[6]. The GRPO diagram keeps the actor and keeps reference weights for an analytical KL term, and swaps the reward model for a deterministic environment, listed as a Python sandbox, unit tests or a matcher ^[11]. Two components go, and the one that mattered for memory is the trainable critic with its optimizer state.

What the post claims DeepSeek-R1 proved is that pure rule-based RL induces deep reasoning behaviour without hand-engineered process reward models ^[12]. That is a claim about math and code, where a program can grade the answer. For memory topology, the post refers readers to the publisher's own NVIDIA GPU Package Architecture Deep Dive ^[13].