Latent Feedback Is a Compute and Data Amplifier
AuthorsBenhao Huang, Chaitanya Dwivedi, Xuezhe Ma, Alexander Moreno, Hector Liu
Affiliations[
Latent feedback may let language models get more value from each training token and FLOP than simply repeating the data.
Key results
Maximum reported compute saving at 400 unique tokens per parameter and two epochs.
Maximum saving predicted by the fitted scaling law.
Unique tokens per parameter at two epochs, approximately the point where latent feedback begins to save compute.
Maximum validation-loss prediction error for the held-out 778M model.
What the paper found
Latent feedback feeds each token’s top-layer state into the next token’s input, giving a Transformer more effective capacity at extra training cost. This study compares it fairly with vanilla Transformers trained for more epochs on the same repeated data, using models from 60M to 778M parameters and deduplicated Dolma 3; the setup uses a Llama 2 tokenizer and OLMo 2 initialization. Once there is enough unique data per parameter, latent feedback reaches a given loss with less compute or less unique data at the same number of epochs. At 400 unique tokens per parameter and two epochs, it saves 44–48% of compute; the fitted scaling law predicts savings of up to 47% in unique data. The compute crossover is about 33 unique tokens per parameter at two epochs: below that, extra training epochs can be the better use of compute. A law fit on models from 60M to 271M parameters predicts held-out 778M validation losses with a maximum error of 0.29%. Downstream results on OLMES broadly agree, though latent feedback is not a universal win when unique data is scarce.
Original abstract
When high-quality data runs out, a common practice is to train on it for several epochs, but each repeat is worth less than the last. Latent feedback, which feeds each token's top-layer state into the next token's input, is another way to spend that compute, yet it has not been tested against a vanilla Transformer that spends the same compute on more epochs. We run this test across a ladder of model sizes, unique data and epochs, tuning both models at every size and amount of unique data. Latent feedback makes every FLOP and every token count for more except when unique data is scarce relative to model size, a regime that today's models are far from. We find that latent feedback serves as an amplifier: with unique data fixed, it reaches a Transformer's loss with less compute; with epochs fixed, it reaches that loss with less unique data. Furthermore, both savings grow with unique data per parameter and with repetition, and downstream tasks agree. To go beyond our grid, we fit one scaling law to both models, in which latent feedback enters as a Transformer with more parameters; it predicts that both savings keep growing with unique data and repetition. Fitted on models from 60M to 271M parameters, the law predicts the losses at a held-out 778M size to within 0.3% and the gap between the two models to within 0.05%. At most, latent feedback saves 48% of the compute and 47% of the unique data. When data, not compute, is the limit, latent feedback is the better use of a FLOP for a model of a given size.
Read the original paperMore in Efficient AI
Browse all 79 papers →MACaQ: Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations
Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang, Lukas Cavigelli
MACaQ aims to make large neural networks cheaper to run by quantizing their weights and activations quickly, with little added inference overhead.
Mobile AI Factory: Building Specialized Language Models for Your Mobile Device
Evan Li, Krishna Deshpande, Wen Cheng, Shyamnath Gollakota
Mobile AI Factory automatically builds and tunes a small language model for a specific task and phone, aiming to deliver strong performance with less compute.
FLAMIA: FLAsh MIcroscaling Attention for Fast and Causally Safe Training on Blackwell
Fedor Velikonivtsev, Mikhail Koltakov, Denis Kuznedelev
FLAMIA makes Transformer attention training faster on Blackwell GPUs by carefully combining low-precision computation with safeguards for causal correctness.