Skip to content
Today
Sign in

Published · 6h agoScience2 min read

Reward-free agent training moves the specification into an LLM judge

GRPO learns only from how a group of sampled runs is ordered, and RULER supplies that ordering with an LLM judge, so what a team maintains is a comparison prompt and the judge model behind it.

2 publishersFor you

Written for builders.See today for builders

Artwork accompanying Reward-free agent training moves the specification into an LLM judge

What happened

  • GRPO samples a group of N completions for each prompt, scores them, computes each one's advantage against the group average, then reinforces the above-average behaviours and suppresses the rest.
  • Only the ordering inside that group drives learning, so scores of 0.3, 0.5 and 0.7 update the model exactly as 30, 50 and 70 would.
  • RULER replaces the hand-written reward function with an LLM judge that reads N full agent trajectories, scores each between 0 and 1, and feeds those scores straight into GRPO.
  • ART's backend runs vLLM for inference and Unsloth-powered GRPO for training, loading a fresh LoRA checkpoint into the inference server after every training step.

Compiled by The ScientistSomething wrong?How this is made

Why it matters

  • decision A team adopting this picks a judge model and writes a comparison prompt, and that pair becomes the task specification to review when the agent starts behaving oddly.
  • constraint Because the update reads only ordering, a judge's ranking error is indistinguishable from a real quality difference, and the GRPO step reinforces whatever came out on top.
  • cost The one-off engineering cost of writing a reward function becomes a recurring inference cost, paid on every group of trajectories at every training step.

Avi Chawla wrote that asking an LLM to "rate this 0-10" produces "inconsistent results", while asking "which of these 4 attempts best achieved the goal?" is "far more reliable" [5][6]. Both prompts return a number. Only the second is a comparison, and a comparison is the form the update consumes [2]. The post does not report a measured comparison of agents trained on judge scores against agents trained on a hand-written reward [15].

The measured results in this record belong to DeepSeekMath. The paper reports 51.7% on the MATH benchmark for DeepSeekMath 7B [11], and 36.2% on MATH plus 64.2% on GSM8K for DeepSeekMath-Base 7B, which beat Minerva 540B [12]. Those are mathematical reasoning benchmarks, not agents ranked by a judge. GRPO's contribution there was largely economic: it forgoes the critic model and estimates the baseline from group scores, which the authors say significantly reduces training resources [10].

What now runs inside the loop is the judge. Every trajectory in the group has to be read and scored, once per training step [14]. Of the six steps in ART's documented loop, one computes a parameter update; a second copies the resulting LoRA checkpoint into the inference server for the next round of rollouts [13][9]. The judge sits at the reward step and never learns [8].

Chawla's framing is that the reward function was the bottleneck, since an email agent needed labeled correct answers and a code agent needed test suites, each its own engineering project [7]. A judge does not remove that work. It moves it into a comparison prompt, and a judge that prefers a tidy failed run to a clumsy success will have GRPO reinforce the failure [1].

What to watch

  • A published comparison of RULER-trained agents against the same task trained on a hand-written reward, with sample sizes reported.
  • Any measurement of how often the judge's ranking of N trajectories agrees with a human ranking of the same runs.
  • Evidence of agents optimising for the judge: trajectories that score well and fail the task.

Claim ledger

Ranked by verification strength, evidence, and original report placement.

  1. [1]

    GRPO works per prompt in four steps: sample a group of N completions from the current model, score each with a reward function, normalize within the group to get each completion's relative advantage against the group average, and update the model to reinforce above-average behaviours and suppress below-average ones.

    ReportedSource: Avi Chawla, blog.dailydoseofds.comView cited source
  2. [2]

    GRPO only needs relative rankings, not absolute scores: whether completions score 0.3, 0.5 and 0.7 or 30, 50 and 70 does not matter, because only the ordering drives learning.

    ReportedSource: Avi Chawla, blog.dailydoseofds.comView cited source
  3. [3]

    RULER (Relative Universal LLM-Elicited Rewards) uses an LLM-as-judge to compare multiple agent trajectories and rank them, with no labeled data required.

    ReportedView cited source

Sources & coverage · 2 publishers

The reporting this story was synthesized from, earliest first. Every link goes to the original.

  1. blog.dailydoseofds.com6h ago
    How to Fine-Tune LLMs in 2026 - by Avi Chawla

    Cited in this coverage: Avi Chawla, blog.dailydoseofds.com

  2. arxiv.org6h ago
    Pushing the Limits of Mathematical Reasoning in Open Language Models