Research on reducing training and inference costs through compression, pruning, quantization, and better computation. Compare savings alongside retained capabilities.
55 papers · Latest edition October 9, 2026
Three of the latest briefs in this collection. Read the evidence and the original papers alongside them.
LoopCD turns the partially computed states of looped Transformers into free guidance, improving accuracy while often cutting inference compute nearly in half.
This work develops scaling laws that explain how looping and sparse experts can be combined to build more capable models with less training and inference compute.
For LLM prefix caches, simple recency may beat fancy eviction rules, especially when workloads follow predictable session patterns.
Newest editions first.
Weihao Liu, Huangjie Zheng, Tianrong Chen, Rohit Dilip, Richard He Bai, Yizhu Jiao, Yuyang Wang, Ruixiang Zhang
LoopCD turns the partially computed states of looped Transformers into free guidance, improving accuracy while often cutting inference compute nearly in half.
Yanbei Chen, Anirudh Goyal, Raghuraman Krishnamoorthi
This work develops scaling laws that explain how looping and sparse experts can be combined to build more capable models with less training and inference compute.
Yiyu Liu, Minlan Yu, Juncheng Yang
For LLM prefix caches, simple recency may beat fancy eviction rules, especially when workloads follow predictable session patterns.
Andrei Panferov, Maximilian Kleinegger, Sweta Priyadarshi, Tijmen Blankevoort, Dan Alistarh
Disaggregated quantization gives LLM prefill and decode their own specialized weights and formats, improving low-bit accuracy while speeding up first-token generation.
Yubo Li, Yidi Miao, Ramayya Krishnan, Rema Padman
A cheap judge handles confident cases while uncertain ones are escalated, preserving nearly all of a powerful LLM judge's accuracy for a fraction of the cost.
Kimia Hamidieh, Giannis Daras, Antonio Torralba
PoEM aims to mix the behaviors of already RL-trained models to predict how they would perform under a new reward, without running expensive RL again.
DeepSeek-AI, :, Anyi Xu, B. Li, Bangcai Lin, Bing Xue, BingCheng Xian, Bingzheng Xu, Bochao Wu, Bowei Zhang, Boyi Deng, C. C. Yu, Chao Jin, Chaofan Lin, Chen Dong, Chenbing Wang, Chenfan Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chengyuan Zhang, Chenhao Xu, Chenqi Zhao, Chenze Shao, Chuhao Wang, Chuqi Zhang, Damai Dai, Dejian Yang, Deli Chen, Di Huang, Di Wu, Donghao Li, Erhang Li, Eric Fu, F. Zhou, Fangwei Zhou, Fangyun Lin, Fangzhou Yuan, Feiyu Xia, Fucong Dai, Guangbo Hao, Guanglin Li, Guanting Chen, Guoai Cao, Guofan Fan, Guolai Meng, Guowei Li, Haichuan Zhang, Haiyang Ma, Haiyang Shen, Han Li, Han Yu, Han Zhang, Hangyuan Deng, Hanwei Xu, Hanxiang Xu, Hanxun Zhong, Hao Guo, Hao Jiang, Hao Li, Hao Qin, Haodong Wen, Haofen Liang, Haofeng Huang, Haohua Liu, Haoling Zhang, Haoming Luo, Haoran Yang, Haotian Xu, Haotian Yuan, Haoting Huang, Haowen Luo, Haoyang Cai, Haoyu Chen, Haozhe Ji, Hengran Zhang, Hengrui Wang, Hengxu Wu, Honghui Ding, Hongxuan Tang, Huadong Wang, Huanqi ...
DeepSeek-V4.1-Flash targets million-token agents by dramatically compressing KV caches while maintaining strong multimodal performance.
Shrey Pandit, Xuan-Phi Nguyen, Yiran Zhao, Shafiq Joty
This work makes trillion-scale, million-token MoE training more feasible by flattening every major GPU memory spike instead of merely reducing the average footprint.
Yu Lin, Yiming Wang, Runyuan Cai, Hanze Liu, Xiaodong Zeng
Edge0 uses predicted expert routing to stream a 35B mixture-of-experts model from SSD fast enough to run on a 24GB consumer machine.
Mostafa Elhoushi, Alex Pretko, Nolan Dey, Bin Claire Zhang, Gavia Gray, Gurpreet Gosal, Abdulrahman Mahmoud, Shane Bergsma, Joel Hestness
Layer dropout can make large language models cheaper to train and faster to run without meaningfully sacrificing accuracy.
Changxin Tian, Kunlong Chen, Jia Liu, Ziqi Liu, Zhiqiang Zhang, Jun Zhou
This work shows how to predict learning rates and batch sizes for increasingly sparse MoE models, making large-scale training more compute-efficient and transferable.
Heng Wang, Jielin Qiu, Wenting Zhao, Cheng Qian, Liangwei Yang, Jiawei Han, Heng Ji, Silvio Savarese, Shelby Heinecke, Huan Wang
Instead of carefully scoring which tokens to keep, this work shows that protecting the prompt and randomly evicting the rest can make long-form LLM reasoning substantially faster.
Mantas Lukauskas
Prompt compression can save tokens in English but silently erase meaning in other languages, so multilingual users need much safer compression budgets.
Zihan Qiu, Zekun Wang, Xiao Li, Yanpeng Li, Yang Xu, Yixuan Wang, Huaqing Zhang, Rui Men, Bochao Mao, Chengruidong Zhang, Fan Zhou, Hao Luo, Haofeng Huang, Haoran Lian, Haoyan Huang, Hongqing Chen, Jianwei Zhang, Jing Xu, Junjie Wang, Langshi Chen, Liangyu Wang, Linlang Jiang, Man Yuan, Minmin Sun, Peng Jin, Siqi Zhang, Siyu Wang, Xingzhang Ren, Yakai Wang, Yi Zhang, Yiming Dong, Yizhong Cao, Yubo Ma, Yunfei Mao, Bo Zheng, Dayiheng Liu
Qwen3.8-Next combines sparse experts, hybrid attention, external memory, and optimizer changes to achieve competitive performance with far less computation and improved training stability.
Kairong Luo, Jiarui Cui, Yaorui Yin, Shengqi Chen, Yiming Yang, Linxiang Gao, Yanmohan Wang, Mingzhe Zhang, Kaiyue Wen, Kaifeng Lyu, Wenguang Chen
Puro-2B shows that a reasonably capable 2B language model can be pretrained from scratch for roughly $5,000 using consumer-grade GPUs and an open recipe.
Shuo Yang, Xiaoze Fan, Melissa Pan, Haocheng Xi, Zhe Wang, Shanlin Sun, Kurt Keutzer, Song Han, Matei Zaharia, Chenfeng Xu, Ion Stoica
FreeToken aims to make frontier-scale mixture-of-experts models practical on ordinary personal computers by dynamically coordinating CPU, GPU, memory, and model state.
Adam Fisch, Shubhendu Trivedi, Fantine Huot, William W. Cohen, Michael Kaisers, Mirella Lapata, Kate Larson, Jacob Eisenstein
A smart router decides when it is worth paying for better model-quality estimates before sending each query to the best specialist.
Zixuan Lan, Yanhong Li, Jiawei Zhou
RMM speeds up LLM inference by selectively skipping less informative parts of matrix multiplications while preserving much of the model's accuracy.
Nathan Godey, Yoav Artzi
One nested language model can serve as several differently sized models, cutting training costs while speeding up speculative decoding.
Zhengpei Hu, Kai Li, Dapeng Fu, Xuechao Zou, Yuanhao Tang, Yue Li, Tengfei Cao, Jianqiang Huang
Hard prompt compression can delete the crucial context that makes surviving answers understandable, but a lightweight restoration step can recover much of the lost accuracy.
Nicholas Lourie, Kyunghyun Cho, Karen Ullrich, Sanae Lotfi
Small models can predict large-model behavior after all—but only when their hyperparameters are tuned carefully enough to reveal the hidden scaling laws.
Yuhang Zhan, Lisi Chen, Shuo Shang
ResKV makes long-context language-model decoding more efficient by compressing KV caches while reconstructing the attention contributions of discarded tokens.
Changwoo Baek, Seungjun Shin, Kyeongbo Kong
RestoreKV makes aggressively compressed LLM KV caches behave more like full caches by adding a compact, context-specific learned restoration cache.
Zhenyu Zhang, Zhichao Cao
TokTier makes agentic LLM serving faster by reusing and repairing tokenization state exactly instead of repeatedly tokenizing long conversation transcripts from scratch.
Haiqian Yang, Yuan Cao
This paper argues that autonomous research systems should be judged not only by what they discover, but also by how efficiently they find it.
Zekun Li, Xiaoyan Cong, Hongyu Li, Zhiyang Dou, Chuan Guo, Abhay Mittal, Sizhe An, Srinath Sridhar
Ms.Forcing accelerates real-time video diffusion by using coarser representations for noisier frames while preserving or improving generation quality.
Sarel Weinberger, Amir Hozez
For coding agents, cutting tokens is not the same as cutting costs—and aggressive compression can even break successful software patches.
Peng Xie
Randomly sampling what an AI forgets lets its serving system tell whether a bad answer came from cache compression or from the model itself.
Kaizhen Tan, Heqing Du, Yang Feng
This paper measures how many bits of data LoRA adapters can secretly store and finds that their memorization depends more on where parameters are placed than on how many there are.
Junsong Chen, Jincheng Yu, Yitong Li, Shuchen Xue, Haozhe Liu, Jingyu Xin, Yuyang Zhao, Tian Ye, Zhangjie Wu, Zian Wang, Daquan Zhou, Ping Luo, Song Han, Enze Xie
SANA-Video 2.0 combines mostly linear attention with occasional softmax anchors to make high-quality long-video generation dramatically faster on a single GPU.
Qingyu Zhang, Qianhao Yuan, Hongyu Lin, Yaojie Lu, Xianpei Han, Le Sun, Xiang Li, Ming Xu, Jiarui Li, Xiuyin Zhao
ShortOPD helps pruned language models regain useful free-form generation by training first on the prefixes they can handle and gradually extending their rollout length.
Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, Yunfan Xiong, Yi Qian, Jiaqi Zhu, Shirong Ma, Xiaokang Zhang, Jiasheng Ye, Qinyu Chen, Chengqi Deng, Jiping Yu, Damai Dai, Zhengyan Zhang, Yixuan Wei, Yixuan Tan, Wenkai Yang, Runxin Xu, Yu Wu, Zhean Xu, Xuanyu Wang, Muyang Chen, Rui Tian, Xiao Bi, Zhewen Hao, Shaoyuan Chen, Huanqi Cao, Wentao Zhang, Anyi Xu, Huishuai Zhang, Dongyan Zhao, Wenfeng Liang
DSpark speeds up LLM inference by generating draft tokens in a more structured way and verifying them adaptively, delivering major serving throughput gains in real production traffic.
Yuchuan Ding, Linfei Li, Lin Zhang, Ying Shen
RaysUp is a fast, lightweight way to turn low-resolution vision foundation model features into sharp, high-resolution maps using geometry-aware ray representations.
Shuhong Zheng, Michael Oechsle, Erik Sandström, Marie-Julie Rakotosaona, Federico Tombari, Igor Gilitschenski
This paper makes 3D visual geometry transformers much faster by cleverly choosing which image tokens to keep, cutting compute by over 85% while preserving accuracy.
Lanxiang Hu, Zhaoxiang Feng, Yulun Wu, Haoran Yuan, Yujie Zhao, Yu-Yang Qian, Bojun Wang, Peng Zhao, Daxin Jiang, Yibo Zhu, Tajana Rosing, Hao Zhang
JetSpec makes large language models decode faster by drafting and verifying better token trees in parallel, pushing speculative decoding to much higher speeds.
Mengmeng Ji, Ravi Shanker Raju, Jonathan Lingjie Li, Chen Wu
LongAttnComp makes very long prompts cheaper to process by learning a smarter way to keep the most useful tokens, helping models reason over 100k+ context with less accuracy loss.
Cristobal Eyzaguirre, Jiajun Wu, Juan Carlos Niebles
This paper makes long video understanding cheaper by replacing quadratic video attention with a stateful linear-time cache that preserves much of the accuracy of full self-attention.
Qian Zhao, Kunlong Chen, Changxin Tian, Zhonghui Jiang, Haitao Zhang, Chaofan Yu, Peijie Jiang, Mingliang Gong, Jia Liu, Ziqi Liu, Zhiqiang Zhang, Jun Zhou
This paper explains why some 4-bit LLM training formats quietly introduce harmful rounding bias, and shows that a more uniform 4-bit recipe can make pretraining more stable and efficient.
Yibin Luo, Shiwei Gao, Huichuan Zheng, Youyou Lu, Jiwu Shu
RoundPipe is a new way to fine-tune huge language models efficiently on cheap GPUs by dynamically rotating work across devices to cut pipeline bubbles and boost throughput.
Yan Wang, Qifan Zhang, Jiachen Yu, Tian Liang, Dongyang Ma, Xiang Hu, Zibo Lin, Chunyang Li, Zhichao Wang, Jia Li, Yujiu Yang, Haitao Mi, Dong Yu
This paper introduces a new way to serve ultra-long-context LLMs by predicting which past tokens actually matter, cutting KV-cache memory drastically while keeping accuracy high.
Yubo Li, Yidi Miao
CONF-KV makes long-context LLMs cheaper by using the model’s own confidence to decide how much memory to keep, helping preserve accuracy while cutting KV-cache use.
Charles O'Neill, Alex Sandomirsky, Harry Partridge, Mudith Jayasekara, Max Kirkby
Still makes long-context language models more practical by compressing their key-value cache in one fast forward pass, preserving performance even at extreme compression ratios.
Luoyuan Zhang
The paper argues that AI agents should stop recomputing the same prompt over and over, and instead buy and reuse precomputed KV caches to make LLM serving much cheaper.
Xunhao Lai, Weiqi Xu, Yufeng Yang, Qiaorui Chen, Yang Xu, Lunbin Zeng, Xiaolong Li, Haohai Sun, Haichao Zhu, Vito Zhang, Pengyu Zhao
MiniMax Sparse Attention makes ultra-long-context LLMs much faster by combining blockwise sparse retrieval with GPU-friendly kernels, delivering major inference speedups at 1M-token scale.
Ang Li, Sean McLeish, Haozhe Chen, Nimit Kalra, Zaiqian Chen, Artem Gazizov, Venkata Anoop Suhas Kumar Morisetty, Bhavya Kailkhura, Harshitha Menon, Zhuang Liu, Brian R. Bartoldson, Tom Goldstein, Sanae Lotfi, Micah Goldblum, Pavel Izmailov
This paper introduces latent context models that compress long prompts into compact representations, making long-context language models faster and more memory-efficient without sacrificing as much quality.
Yanbei Chen, Hanxian Huang, Ernie Chang, Jacob Szwejbka, Digant Desai, Zechun Liu, Vikas Chandra, Raghuraman Krishnamoorthi
MobileMoE shows that sparse mixture-of-experts language models can run efficiently on phones, delivering better speed-memory tradeoffs than dense on-device LLMs.
Song Bian, Tao Yu, Shivaram Venkataraman, Youngsuk Park
This paper shows how to use scaling laws plus architecture choices like attention design to build language models that are both cheaper to run and more accurate.
Jianuo Huang, Yaojie Zhang, Qituan Zhang, Hao Lin, Hanlin Xu, Linfeng Zhang
Domino speeds up LLM generation by using a parallel draft model plus a lightweight causal refinement step, delivering faster speculative decoding without sacrificing too much draft quality.
Haodi Lei, Yafu Li, Haoran Zhang, Shunkai Zhang, Qianjia Cheng, Xiaoye Qu, Ganqu Cui, Bowen Zhou, Ning Ding, Yun Luo, Yu Cheng
Draft-OPD improves speculative decoding by training draft models on the kinds of errors they actually make at inference time, yielding faster LLM generation with better acceptance rates.
Yiqun Liu, Yingsheng Wu, Ruqi Yang, Enrong Zheng, Honglei Qiu, Sijun He, Tai Liang, Jingjing Wu, Yuhan Zhou, Yiwei Zhang, Dongyan Chen, Weihan Yi, Xinqi Li, Siqi Bao
PassNet reframes LLM-based compiler optimization as graph pass generation, backing it with a large dataset and benchmark that show LLMs can meaningfully speed up real-world workloads.
Pál András Papp, Aleksandros Sobczyk, Anastasios Zouzias
This paper makes attention much cheaper to run by reducing memory traffic, bringing it closer to the theoretical limits of efficiency.
Zili Zhang, Chengxu Yang, Shenglong Zhang, Chenyu Wang, Yufan Zhang, Tuo Dai, Zhouyang Li, Yuhong Ge, Chao Jin, Xin Jin, Yuliang Liu
BigMac is a new multimodal LLM training pipeline that claims to break the usual tradeoff between speed and memory, making training both faster and more memory-friendly.
Xintong Yang, Hao Gu, Binxing Xu, Lujun Li, Bei Liu, Jiacheng Liu, Qiyuan Zhu, Sirui Han, Yike Guo
IndexMem makes long-context LLMs more efficient by learning what to keep in memory and compressing what gets evicted so the model can still recall important information later.
Junhyuck Kim, Jihun Yun, Haechan Kim, Gyeongman Kim, Joonghyun Bae, Jaewoong Cho
This paper shows how to turn large mixture-of-experts language models into smaller dense models that keep much of the performance while being easier and cheaper to deploy.
Christian Janos Lebeda, David Erb, Tudor Cebere, Aurélien Bellet
Lumberjack makes private random forests much more practical by using a clever heavy-hitter pruning trick that keeps deeper trees useful without leaking sensitive data.