Principled Analysis of Deep Reinforcement Learning Evaluation and Design Paradigms

Ezgi Korkmaz’s paper challenges a central assumption in deep reinforcement learning: that algorithms ranking highly after extensive training will rank highly when data are scarce. Its theory proves a non-monotonic relationship between performance and sample complexity: lower-capacity models with larger inherent Bellman error can outperform more expressive distributional methods in the low-data regime, while the ordering reverses asymptotically. Large-scale experiments in the Arcade Learning Environment compare the 100K benchmark with 200M-frame training, revisiting the Atari research tradition associated with DeepMind, including DQN, C51, QRDQN, IQN, and the dueling architecture. In ALE 100K, dueling achieves a human-normalized median score of 0.2304, versus 0.0941 for C51, 0.0820 for QRDQN, and 0.0528 for IQN, showing that higher-capacity value-distribution models can be disadvantaged by limited data. The paper also exposes evaluation bias in data-efficient reinforcement learning: DRQ’s original comparisons reported an 82% gain over DER, but direct comparison with the core dueling baseline reduces that gain to 11%, while one reproduction performs 15% below dueling. Korkmaz argues that benchmarks must include core algorithms, match methods across capacity and data regimes, report direct baselines, and scrutinize dataset-selection assumptions, because conclusions drawn from ALE 100K can systematically misdirect subsequent research.