HyperThink: Text-to-Parameter Hypernetworks for Efficient Reasoning
AuthorsDonggyun Kim, Jack Lu, Chanwoo Kim, Mengye Ren, Seunghoon Hong
AffiliationsKAIST School of Computing · NYU Courant Institute School of Mathematics, Computing, and Data Science · Oct 2026
HyperThink tries to make LLM reasoning faster by turning each question into a small, temporary parameter update instead of a long chain-of-thought.
Key results
The hypernetwork predicts updates to 0.015% of the model’s parameters.
Each layer-wise codebook constrains updates to reusable discrete patterns.
HyperThink with Qwen3-0.6B, compared with 46.84% for native non-thinking.
HyperThink with Qwen3-0.6B; full thinking uses 4,535.70G.
Approximate latency as a fraction of full-thinking latency on question-answering benchmarks.
What the paper found
HyperThink replaces long, sequential thinking traces—used by reasoning systems such as OpenAI o1 and DeepSeek-R1—with a single query-conditioned parameter update. A lightweight hypernetwork reads each question and predicts shifts to just 90,122 bias parameters, or 0.015% of Qwen3-0.6B’s parameters; a 256-entry vector-quantized codebook encourages reuse of robust update patterns, while the base model stays frozen. Trained on the model’s own thinking-mode responses, HyperThink then generates a concise step-by-step answer without an intermediate trace. On MATH-500 with Qwen3-0.6B, it achieves 48.76% accuracy at 1,150.83G FLOPs, compared with 46.84% at 962.35G for native non-thinking and 52.64% at 4,535.70G for full thinking. The gains are strongest in the low-latency regime, and experiments with SmolLM3-3B and Olmo-3-7B-Think extend the evaluation to code, logic, and question answering; on Olmo-3-7B-Think’s question-answering benchmarks, HyperThink operates at approximately 7–9% of full-thinking latency. The results suggest that discrete, query-specific parameter steering can preserve some reasoning benefits without paying the decoding cost of long traces.
Original abstract
Long-form thinking traces can substantially improve the multi-step reasoning performance of large language models (LLMs), but they introduce high inference-time overhead, with latency dominated by sequential decoding. We propose HyperThink, a text-to-parameter approach that amortizes this reasoning computation into a single query-conditioned parameter update: a lightweight hypernetwork reads the question and predicts updates to a small subset of the base LLM's parameters, while a vector-quantized decoder constrains them to a finite set of reusable patterns to improve robustness and transfer. Trained end-to-end on outputs from the base model itself, HyperThink eliminates long thinking traces at test time: after one hypernetwork forward pass, the adapted model generates a concise step-by-step solution and final answer without an intermediate trace, using far fewer tokens while retaining strong reasoning performance. Empirically, HyperThink improves the low-latency region of the accuracy-latency trade-off on mathematical and general reasoning tasks, with its strongest gains in the near-non-thinking regime.
Read the original paperMore in Efficient AI
Browse all 64 papers →DLoop: Looped Speculative Decoding
Geonmo Gu, Byeongho Heo, HeeJae Jun, Yoohoon Kang, Sangmin Lee, Sangdoo Yun, Dongyoon Han
DLoop speeds up large language model generation by letting draft models propose several batches of tokens before the expensive target-model verification step.
The Surprising Effectiveness of Shared Memory in Looped Transformers
Giovanni Monea, Keshav Ramji, Yousef El-Kurdi, Luis A. Lastras, Yoav Artzi, Nathan Godey, Ramón Fernandez Astudillo
A clever memory-sharing scheme lets repeatedly applied Transformers use far less inference memory while performing even better.
FrugalEvo: Towards Cost-Aware LLM-Guided Program Evolution
Hui Chen, Xuan Qi, James Xu Zhao, Zhaopeng Feng, Shilong Liu, Kuang Xu, Pang Wei Koh, Bryan Hooi
FrugalEvo makes LLM-driven program evolution more practical by combining stronger strategic models, cheaper implementers, and cache-aware search to achieve competitive results at far lower cost.