Library of Layers (LoL): Adaptively Ordering Transformer Layers at Runtime
AuthorsAbdellah Rahmani, Sean McLeish, Ang Li, Sanae Lotfi, Pascal Frossard, Micah Goldblum
AffiliationsEPFL · New York University · Columbia University
LoL lets a transformer choose the order of its own layers at inference time, aiming for more flexible and efficient computation.
Key results
Scale of the model used for multilingual and routing-control experiments.
Token count for the multilingual model and synthetic strategy experiments.
Sequence-level router-feature probe distinguishing safe from unsafe text.
LoL score in the 1B-parameter math-data evaluation.
LoL score in the 1B-parameter math-data evaluation.
What the paper found
Library of Layers, or LoL, lets a transformer choose which block to run next instead of executing every layer in a fixed order. At each routed depth, a top-1 router selects a block from a shared library for each token window, allowing blocks to be reused, reordered, or left unused. Its strided window attention shifts boundaries between depths, preserving causal access while passing information between neighboring windows. Changing the library size moves LoL between parameter reuse, like recurrent transformers, and sparse computation, like mixture-of-experts models; gradually growing the routed depth during training reduces early compute. In synthetic Sudoku and maze tasks, the learned routes represent distinct solving strategies, and forcing a route at inference can steer the model toward that strategy. A 360M-parameter model trained on 10B tokens across 10 languages also learned language-dependent routing, while a probe of sequence-level router features separated safe from unsafe text with 0.95 AUROC. In math-data experiments, LoL remained competitive with dense baselines: it scored 50.3 on ARC-Easy and 59.7 on PIQA. The 1B-parameter configuration follows the model shape of Meta’s Llama 3.2 1B. The central contribution is a single architecture that makes layer order itself an input-dependent, inspectable, and steerable part of computation.
Original abstract
Computer programmers chain together the same reusable computational primitives, for example, conditional statements or loops, but in different orders to implement different algorithms. In contrast, existing language model architectures always execute their layers in the same order. We introduce Library of Layers (LoL), an architecture that dynamically orders its layers during inference to solve the problem at hand. LoL has a single hyperparameter, the size of its global library of transformer blocks, that allows it to transition seamlessly between parameter reuse and sparsity, generalizing the parameter-efficiency and FLOPs-efficiency tradeoffs of dense, depth-recurrent, and mixture of experts architectures in one unified family. We show that LoL achieves competitive performance with all three above baselines when pretrained on text data. On synthetic sudoku and maze data, we observe that the model employs distinct layer orderings to implement different solution strategies, and we can intervene on routing to force a model to apply a specific solution strategy to a problem instance. In a multilingual natural language setting, we find that LoL chooses different layer order patterns based on language and script. Similarly, we train LoL on safety data and fit probes to its router outputs, finding that these probes distinguish safe from unsafe text with high AUROC. LoL provides a path toward a future in which models dynamically compose their own computational graph, enabling more efficient and controllable reasoning.
Read the original paperMore in Efficient AI
Browse all 79 papers →Latent Feedback Is a Compute and Data Amplifier
Benhao Huang, Chaitanya Dwivedi, Xuezhe Ma, Alexander Moreno, Hector Liu
Latent feedback may let language models get more value from each training token and FLOP than simply repeating the data.
MACaQ: Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations
Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang, Lukas Cavigelli
MACaQ aims to make large neural networks cheaper to run by quantizing their weights and activations quickly, with little added inference overhead.
Mobile AI Factory: Building Specialized Language Models for Your Mobile Device
Evan Li, Krishna Deshpande, Wen Cheng, Shyamnath Gollakota
Mobile AI Factory automatically builds and tunes a small language model for a specific task and phone, aiming to deliver strong performance with less compute.