Copy-Paste Language Models
AuthorsNathan Godey, Yoav Artzi
AffiliationsCornell University · New York City, USA
Copy-Paste Language Models add a low-overhead copying mechanism that improves language-model training and performance, especially in long-context and retrieval settings.
Key results
Maximum training-token budget used for models up to 1.7B parameters.
Up to this many times faster to reach the vanilla model's final loss.
Times less memory for a sliding-window CPLM matching full attention.
Average across the tested context-length and needle-depth grid, versus 14.3% for vanilla.
What the paper found
Copy-Paste Language Models add a lightweight pointer head alongside the standard language-model head. It attends to earlier token positions, while learned <copy> and <sink> gates decide whether to copy or fall back to ordinary prediction; training uses the usual next-token loss, with no copy labels or auxiliary objective. The key result is that this second prediction route also gives the backbone a less-compressed gradient signal, easing the bottleneck created by the vocabulary projection. Across models from 135M to 1.7B parameters, pretrained on FineWeb-Edu with up to 200B tokens, CPLMs improve perplexity and downstream performance across full-attention, sliding-window, hybrid, and Gated DeltaNet backbones. They reach vanilla models’ final loss up to 2× faster, and a sliding-window CPLM matches full attention with 12× less memory per token. On a needle-retrieval test spanning 1k to 64k tokens, the SW-256 CPLM averages 99.4% accuracy, compared with 14.3% for its vanilla counterpart. The paper also reports gains after supervised fine-tuning on math, code, and instruction-following tasks, evaluated in part with Tulu-3 and NVIDIA’s Nemotron-3-SFT-Math-v4 data. The copy head adds little overhead, and experiments suggest its benefit comes mainly from improving how the backbone learns, not simply from copying tokens at inference.
Original abstract
Transformers are intuitively considered to be efficient at copying from their inputs. We actually show that the ability of LLMs to copy in context can be improved through explicit modeling. We propose Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head to the conventional language modeling (LM) head. CPLMs improve training efficiency, downstream performance, and offer new operating points in long-context and retrieval setups. We validate our approach by pretraining models from 135M to 1.7B parameters, showing improvements across all scales and backbones with negligible training and inference overheads. We explain the success of CPLMs by showing that they partially mitigate the LM head gradient bottleneck through an alternative gradient path.
Read the original paperMore in Large Language Models
Browse all 87 papers →Large Flow Language Models
Arnav Chavan, Chanhyuk Lee, Udbhav Bamba, Jaehoon Yoo, Seunghoon Hong, Aditi Raghunathan, Jinwoo Kim
Lafa turns large language models into fast continuous-flow generators, reporting up to 2.54× throughput at comparable quality.
Retrieval Augmented Generation Compositionality
Ido Levy, Segev Shlomov, Leshem Choshen
A large-scale study finds that RAG techniques that work well alone may not work well together, and the best combination can change as the knowledge corpus grows.
HuatuoGPT-3: RL-Only Domain Adaptation from Base Models
Junying Chen, Xinyuan Xie, Ziniu Li, Wenyuan Gu, Jianquan Li, Xiang Wan, Guangjun Yu, Ruoyu Sun, Haizhou Li, Benyou Wang
OnePO uses temporary teacher guidance and reinforcement learning alone to turn general LLMs into stronger medical specialists without the usual supervised fine-tuning stage.