NTH

Library of Layers (LoL): Adaptively Ordering Transformer Layers at Runtime

AuthorsAbdellah Rahmani, Sean McLeish, Ang Li, Sanae Lotfi, Pascal Frossard, Micah Goldblum

AffiliationsEPFL · New York University · Columbia University

November 11, 2026 2 min read
Watch on YouTube
The one-line take

LoL lets a transformer choose the order of its own layers at inference time, aiming for more flexible and efficient computation.

Key results

360M
Model size

Scale of the model used for multilingual and routing-control experiments.

10B
Training tokens

Token count for the multilingual model and synthetic strategy experiments.

0.95
Safety probe AUROC

Sequence-level router-feature probe distinguishing safe from unsafe text.

50.3
ARC-Easy

LoL score in the 1B-parameter math-data evaluation.

59.7
PIQA

LoL score in the 1B-parameter math-data evaluation.

What the paper found

Library of Layers, or LoL, lets a transformer choose which block to run next instead of executing every layer in a fixed order. At each routed depth, a top-1 router selects a block from a shared library for each token window, allowing blocks to be reused, reordered, or left unused. Its strided window attention shifts boundaries between depths, preserving causal access while passing information between neighboring windows. Changing the library size moves LoL between parameter reuse, like recurrent transformers, and sparse computation, like mixture-of-experts models; gradually growing the routed depth during training reduces early compute. In synthetic Sudoku and maze tasks, the learned routes represent distinct solving strategies, and forcing a route at inference can steer the model toward that strategy. A 360M-parameter model trained on 10B tokens across 10 languages also learned language-dependent routing, while a probe of sequence-level router features separated safe from unsafe text with 0.95 AUROC. In math-data experiments, LoL remained competitive with dense baselines: it scored 50.3 on ARC-Easy and 59.7 on PIQA. The 1B-parameter configuration follows the model shape of Meta’s Llama 3.2 1B. The central contribution is a single architecture that makes layer order itself an input-dependent, inspectable, and steerable part of computation.

Original abstract

Computer programmers chain together the same reusable computational primitives, for example, conditional statements or loops, but in different orders to implement different algorithms. In contrast, existing language model architectures always execute their layers in the same order. We introduce Library of Layers (LoL), an architecture that dynamically orders its layers during inference to solve the problem at hand. LoL has a single hyperparameter, the size of its global library of transformer blocks, that allows it to transition seamlessly between parameter reuse and sparsity, generalizing the parameter-efficiency and FLOPs-efficiency tradeoffs of dense, depth-recurrent, and mixture of experts architectures in one unified family. We show that LoL achieves competitive performance with all three above baselines when pretrained on text data. On synthetic sudoku and maze data, we observe that the model employs distinct layer orderings to implement different solution strategies, and we can intervene on routing to force a model to apply a specific solution strategy to a problem instance. In a multilingual natural language setting, we find that LoL chooses different layer order patterns based on language and script. Similarly, we train LoL on safety data and fit probes to its router outputs, finding that these probes distinguish safe from unsafe text with high AUROC. LoL provides a path toward a future in which models dynamically compose their own computational graph, enabling more efficient and controllable reasoning.

Read the original paper

More in Efficient AI

Browse all 79 papers →
01Efficiency

Latent Feedback Is a Compute and Data Amplifier

Benhao Huang, Chaitanya Dwivedi, Xuezhe Ma, Alexander Moreno, Hector Liu

Latent feedback may let language models get more value from each training token and FLOP than simply repeating the data.

Read analysis