NTH

FewStepRL: Fine-tuning Few-step Diffusion Models with Reinforcement Learning

AuthorsHaitao Lin, Minsi Ren, Yang Li, Xingtong Ge, Zheyuan Hu, Peiyan Hu, Tailin Wu

AffiliationsTencent Hunyuan · Westlake University · University of Chinese Academy of Sciences

November 17, 2026 3 min read
Watch on YouTube
The one-line take

FewStepRL adapts multiple reinforcement-learning fine-tuning methods to make reward-driven improvements to fast, few-step diffusion models.

Key results

0.465
ReFL GenEval2

Best reported GenEval2 score in the mixed-reward image experiment.

1.641
FewStepOPSD mixed training reward

Final reward, increased from 1.450 in the image experiment.

0.945
MeanFlowNFT OCR accuracy

Highest logged OCR accuracy in the OCR setting.

83.70
MeanFlowAWM VBench overall

Best VBench overall score in the video experiment.

219
ReFL iteration time

Seconds per iteration in the mixed-reward image experiment.

What the paper found

FewStepRL tackles a mismatch in diffusion-model fine-tuning: few-step generators predict average velocity across an interval, while many reinforcement-learning objectives assume instantaneous velocity. It converts objectives into a shared clean-sample, or x-space, view, decomposes their gradients into self-regression and reward-augmentation terms, then adapts FlowGRPO, AWM, DiffusionNFT, DiffusionOPSD, and ReFL by defining how their training anchors are computed. The experiments fine-tune AnyFlow-distilled Stability AI SD3.5-Medium for images and Wan2.1-T2V-1.3B for video, including runs on NVIDIA H20 GPUs. For differentiable image rewards, reward-gradient methods perform strongly: ReFL reaches 0.465 on GenEval2, while FewStepOPSD raises the mixed training reward from 1.450 to 1.641. When gradients are unavailable, DiffusionNFT-style methods are effective; MeanFlowNFT reaches 0.945 OCR accuracy, though reward gains can coincide with declines in held-out quality. On video, MeanFlowAWM records 83.70 VBench overall, close to the 83.68 starting score. ReFL takes 219 seconds per iteration in the image experiment. Finally, sampling with more steps brings no consistent aggregate quality improvement, even though selected images look better with two to four times the training step count.

Original abstract

Reinforcement learning (RL) has become a standard post-training method for diffusion models, but most RL objectives roll out the current policy at every iteration, and with a many-step sampler these rollouts typically dominate the cost of training. Few-step models cut this cost, yet they model an average velocity over an interval, whereas existing RL objectives are built for an instantaneous velocity with a many-step sampler. We adapt RL objectives to few-step models systematically in three phases, namely a transformation, a decomposition and an adaptation. The transformation to $x$-space expresses each objective we consider through a common policy prediction, at the clean end by default and at an explicitly marked level for transition or flow-map objectives, so that objectives stated over densities, velocities or rewards become comparable. The decomposition writes the gradient of each of these objectives as the sum of a self-regression toward a reference and a reward augmentation, so that a method reduces, at the gradient level, to a choice of anchors and weights. The adaptation computes each anchor from a few-step model at the end of its transport or along its tangent, with the instantaneous velocity recovered from the network itself or from an auxiliary network. This yields few-step versions of FlowGRPO, AWM, DiffusionNFT, DiffusionOPSD and ReFL in one modular codebase. In our experiments, reward-gradient-based methods are the most effective when the reward is differentiable and its gradient is affordable; otherwise DiffusionNFT-style methods generally reach high rewards in relatively few iterations despite a sometimes higher per-iteration cost. Extra inference steps show no systematic aggregate benefit in our experiments, although selected outputs visibly improve with two to four times as many sampling steps as in training.

Read the original paper

More in Diffusion Models

Browse all 82 papers →