Calibrated Decision Rewards: Training Language Models with a Judge That Knows How Sure It Is
arXiv — cs.AI ·
What to know
The paper introduces RLCDR, a reinforcement learning method that uses a calibrated decision model as a reward signal, converting an LLM into a probability distribution over typed questions about a response. Training Qwen3-1.7B with GRPO on MATH, this reference-free reward improves MATH500 accuracy from 72.2% to 76.8% (+4.65 points, p < 1e-4), matching training on the answer key (76.9%), while a random reward gains only +0.65. The reward remains calibrated on the policy's own samples, with expected calibration error half that of a softmax judge on a 27B backbone (0.074 vs. 0.141), and shows no sign of reward hacking.
Source and evidence
arXiv — cs.AIOriginal language · und ·Original author: Rudransh Agnihotri, Manasvi Kapoor · Source: arXiv and the named paper authors. Descriptive metadata: CC0 1.0.Source licence
Why it matters by industry
- Artificial intelligence
This research introduces a novel reward modeling approach for reinforcement learning in language models, directly relevant to the 'Reasoning' and 'Models' subtopics, with potential implications for improving model training efficiency and reliability.
Reviewed summary. The complete article remains at its original source.