Calibrated decision rewards improve language model training without answer keys
A new reinforcement learning method uses a probability-based judge model to score language model outputs without needing a reference answer.
2 min read, Saturday, 21 November

Training language models effectively often requires knowing whether an answer is correct. While exact answer checkers work well, they require a reference answer for every single problem. Learned reward models and large language model judges can evaluate open-ended tasks, but their scores rarely represent true probabilities, making it difficult to gauge how certain the judge actually is about its decision.
Researchers have introduced a technique called Reinforcement Learning from Calibrated Decision Rewards, or RLCDR. This method converts a language model into a decision model that answers specific, typed questions about a response. Instead of a simple text output, the model produces a probability distribution in a single forward pass. These probabilities are debiased over option orders and calibrated using a small set of about one hundred labels.
Key facts
- Method
- RLCDR (Reinforcement Learning from Calibrated Decision Rewards)
- Model
- Qwen3-1.7B
- Baseline Accuracy
- 72.2%
- Improved Accuracy
- 76.8%
- Answer Key Accuracy
- 76.9%
- Authors
- Rudransh Agnihotri and 1 other authors
- MATH500 Accuracy
- Base Model: 72.2%
- RLCDR: 76.8%
- Answer Key: 76.9%
- Random Reward: —
| Metric | Base Model | RLCDR | Answer Key | Random Reward |
|---|---|---|---|---|
| MATH500 Accuracy | 72.2% | 76.8% | 76.9% | — |
Training performance and accuracy
To test the approach, researchers trained a Qwen3-1.7B model using Group Relative Policy Optimization on the MATH dataset. Using this reference-free reward raised the accuracy on the MATH500 benchmark from 72.2 percent to 76.8 percent. This improvement of 4.65 points nearly matched the performance of training directly on the official answer key, which achieved 76.9 percent. A random reward, by comparison, gained only 0.65 points.
The performance gains were most noticeable on harder problems, where accuracy on the most difficult level rose from 44.8 percent to 51.7 percent. Furthermore, the reward signal remained reliable even as the policy's own samples changed during training. Its expected calibration error was half that of a standard softmax judge running on the same 27B backbone, measuring 0.074 compared to 0.141, and the model showed no signs of reward hacking.
Finding errors in reasoning
Beyond scoring final answers, the calibrated decision model excels at identifying mistakes within multi-step reasoning. Without any specialized step-by-step training, the model successfully located the first incorrect step in a solution better than a dedicated 7B process reward model, achieving a ProcessBench F1 score of 0.772 against 0.722 across 20 out of 20 splits.
However, the study also mapped the limits of the approach. When trained exclusively on problems that the base model mostly fails to solve, the reference-free reward still improved the policy by 1.9 MATH500 points, but it lagged behind the answer key. On this specific distribution, the model's probabilities became overconfident. The researchers stated six hypotheses in advance and tested each with a control design to ensure rigorous validation.
Source: Original announcement
Worth a read?



Comments