Discretizing Reward Models

Discretizing Reward Models, from Carnegie Mellon University and Meta Superintelligence Labs, argues that continuous reward models can be dangerously oversensitive: they often assign different scores to responses that are equally useful, which creates exploitable reward gradients in RLHF. The paper separates reward quality into discriminative ability and specificity, showing that popular models such as Skywork V1, Skywork V2, GRM, and ArmoRM can score highly on standard benchmarks yet still fail this tie-sensitive criterion. The core result is that a training-free discretization method, reward clustering, uses Monte Carlo dropout to estimate reward uncertainty, hierarchically clusters response scores, and maps clusters to ordinal bins; in the theory, an optimal midpoint threshold can preserve perfect discriminative ability while eliminating oversensitivity in the binary case. On RewardBench 2’s Ties subset, clustered rewards improve the average of discriminative ability and specificity for all four reward models, with Skywork V1 rising from 70.8 to 74.9 and GRM from 69.2 to 80.6. In a mixed-reward IFEval simulation, discretization suppresses overoptimization of spurious hedging rewards, and in a verifier-free multi-task RL setup on 30K prompts each from RLVR-IFeval, RLVR-MATH, RLVR-GSM, and WildChat, discretized rewards were never significantly worse than raw rewards and often much better. The method adds only a modest 15% runtime overhead during GRPO training, while using 8 H100 GPUs and 4 dropout samples.