FewStepRL: Fine-tuning Few-step Diffusion Models with Reinforcement Learning
AuthorsHaitao Lin, Minsi Ren, Yang Li, Xingtong Ge, Zheyuan Hu, Peiyan Hu, Tailin Wu
AffiliationsTencent Hunyuan · Westlake University · University of Chinese Academy of Sciences
FewStepRL adapts multiple reinforcement-learning fine-tuning methods to make reward-driven improvements to fast, few-step diffusion models.
Key results
Best reported GenEval2 score in the mixed-reward image experiment.
Final reward, increased from 1.450 in the image experiment.
Highest logged OCR accuracy in the OCR setting.
Best VBench overall score in the video experiment.
Seconds per iteration in the mixed-reward image experiment.
What the paper found
FewStepRL tackles a mismatch in diffusion-model fine-tuning: few-step generators predict average velocity across an interval, while many reinforcement-learning objectives assume instantaneous velocity. It converts objectives into a shared clean-sample, or x-space, view, decomposes their gradients into self-regression and reward-augmentation terms, then adapts FlowGRPO, AWM, DiffusionNFT, DiffusionOPSD, and ReFL by defining how their training anchors are computed. The experiments fine-tune AnyFlow-distilled Stability AI SD3.5-Medium for images and Wan2.1-T2V-1.3B for video, including runs on NVIDIA H20 GPUs. For differentiable image rewards, reward-gradient methods perform strongly: ReFL reaches 0.465 on GenEval2, while FewStepOPSD raises the mixed training reward from 1.450 to 1.641. When gradients are unavailable, DiffusionNFT-style methods are effective; MeanFlowNFT reaches 0.945 OCR accuracy, though reward gains can coincide with declines in held-out quality. On video, MeanFlowAWM records 83.70 VBench overall, close to the 83.68 starting score. ReFL takes 219 seconds per iteration in the image experiment. Finally, sampling with more steps brings no consistent aggregate quality improvement, even though selected images look better with two to four times the training step count.
Original abstract
Reinforcement learning (RL) has become a standard post-training method for diffusion models, but most RL objectives roll out the current policy at every iteration, and with a many-step sampler these rollouts typically dominate the cost of training. Few-step models cut this cost, yet they model an average velocity over an interval, whereas existing RL objectives are built for an instantaneous velocity with a many-step sampler. We adapt RL objectives to few-step models systematically in three phases, namely a transformation, a decomposition and an adaptation. The transformation to $x$-space expresses each objective we consider through a common policy prediction, at the clean end by default and at an explicitly marked level for transition or flow-map objectives, so that objectives stated over densities, velocities or rewards become comparable. The decomposition writes the gradient of each of these objectives as the sum of a self-regression toward a reference and a reward augmentation, so that a method reduces, at the gradient level, to a choice of anchors and weights. The adaptation computes each anchor from a few-step model at the end of its transport or along its tangent, with the instantaneous velocity recovered from the network itself or from an auxiliary network. This yields few-step versions of FlowGRPO, AWM, DiffusionNFT, DiffusionOPSD and ReFL in one modular codebase. In our experiments, reward-gradient-based methods are the most effective when the reward is differentiable and its gradient is affordable; otherwise DiffusionNFT-style methods generally reach high rewards in relatively few iterations despite a sometimes higher per-iteration cost. Extra inference steps show no systematic aggregate benefit in our experiments, although selected outputs visibly improve with two to four times as many sampling steps as in training.
Read the original paperMore in Diffusion Models
Browse all 82 papers →JVT: Just Video Transformer makes pixel-space video diffusion simple
Jianjun Zhou, Hua Geng, Haoyu Guo
JVT shows that a straightforward Transformer can generate video directly in pixels—and even produce matching depth-like geometry alongside it.
Restoring Diversity in Distilled Image and Video Generators by Probing Activations
Furkan Horoz, Ozgur Kara, Kaan Oktay, Adil Kaan Akan, Atakan Kara, Anand Bhattad, James Matthew Rehg
The paper restores variety in fast distilled image and video generators by finding and correcting a collapse in their internal activation directions.
BYOD: Build Your Own DLM - Efficient Autoregressive-to-Diffusion Language Model Conversion
Ruurd J. A. Kuiper, Robert A. Bagheri
BYOD shows how to turn existing language models into diffusion generators with small LoRA adapters, while highlighting the quality gap that remains.