DLoop: Looped Speculative Decoding
AuthorsGeonmo Gu, Byeongho Heo, HeeJae Jun, Yoohoon Kang, Sangmin Lee, Sangdoo Yun, Dongyoon Han
AffiliationsNAVER AI Lab NAVER AI Search Platform Korea University
DLoop speeds up large language model generation by letting draft models propose several batches of tokens before the expensive target-model verification step.
Key results
Reported range across the evaluated draft-model settings.
Reported range across the evaluated draft-model settings.
Average speedup rises from 3.77× to 4.76× with DLoop.
Average speedup with tree verification and DLoop.
User prompts used to train the loop-aware draft models.
What the paper found
DLoop changes speculative decoding by letting a draft model produce several blocks before the large target model verifies them together. After each block, a confidence gate sums the draft model’s token log probabilities and continues drafting if the score clears a threshold. Loop-aware training teaches the drafter to continue using its own hidden states for tokens the target model has not yet verified. The method preserves lossless decoding, adds no parameters, and applies to both parallel and autoregressive drafters. Across EAGLE-3, DFlash, Domino, DSpark, and multi-token prediction modules evaluated with Alibaba’s Qwen3 and Google’s Gemma models, reported wall-clock speedup rises 5–41%, while average acceptance length rises 12–102%. For DSpark with Qwen3-4B, average speedup increases from 3.77× to 4.76×; with tree verification, DFlash paired with DDTree reaches 6.48×. Loop-aware training used 1.3 million PerfectBlend prompts. The central trade-off is deliberate: spend extra lightweight drafting passes to avoid costly target-model verification passes.
Original abstract
Speculative decoding accelerates autoregressive generation in large language models. In each drafting stage, a lightweight draft model proposes tokens that the target model subsequently verifies. With increasingly capable draft models, we find that the target model frequently accepts all tokens produced in a drafting stage. A verification nevertheless follows each drafting stage, resulting in unnecessary target-model forward passes even when drafting could have continued. Adaptive draft length methods decide during decoding how many draft tokens precede a verification, but they raise the speedup only for autoregressive draft models. For a parallel draft model, drafting further requires target-model hidden states for draft tokens that have not been verified. We propose DLoop, a looped form of speculative decoding that adaptively performs multiple drafting stages before verification. DLoop continues drafting while the draft model remains confident and verifies all accumulated draft tokens together. Loop-aware training keeps the draft model reliable in the additional drafting stages by exposing it to its own hidden states for unverified draft tokens. By spending additional draft-model forward passes, DLoop reduces the number of target-model forward passes required for verification. Across diverse speculative decoding methods including EAGLE-3, DFlash, Domino, DSpark, and multi-token prediction modules, DLoop improves the wall-clock speedup by 5 to 41 percent while preserving lossless decoding. Code will be available at https://github.com/naver-ai/DLoop.
Read the original paperMore in Efficient AI
Browse all 64 papers →HyperThink: Text-to-Parameter Hypernetworks for Efficient Reasoning
Donggyun Kim, Jack Lu, Chanwoo Kim, Mengye Ren, Seunghoon Hong
HyperThink tries to make LLM reasoning faster by turning each question into a small, temporary parameter update instead of a long chain-of-thought.
The Surprising Effectiveness of Shared Memory in Looped Transformers
Giovanni Monea, Keshav Ramji, Yousef El-Kurdi, Luis A. Lastras, Yoav Artzi, Nathan Godey, Ramón Fernandez Astudillo
A clever memory-sharing scheme lets repeatedly applied Transformers use far less inference memory while performing even better.
FrugalEvo: Towards Cost-Aware LLM-Guided Program Evolution
Hui Chen, Xuan Qi, James Xu Zhao, Zhaopeng Feng, Shilong Liu, Kuang Xu, Pang Wei Koh, Bryan Hooi
FrugalEvo makes LLM-driven program evolution more practical by combining stronger strategic models, cheaper implementers, and cache-aware search to achieve competitive results at far lower cost.