Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference

This paper revisits layer dropout, or stochastic depth, for modern large language models, challenging the practice seen in dense pretraining recipes associated with GPT-3, PaLM, and Meta’s Llama 3. Instead of dropping individual activations, the method skips entire transformer blocks per sequence, creating hardware-efficient structured sparsity. Across more than 2400 training experiments covering 271M to 8.2B-parameter models and datasets up to 160B tokens, the authors find that accuracy depends critically on configuration: dropout should increase with depth through Increasing Layer Distribution, then decrease linearly to zero over training through the Decreasing Time Schedule, with residual scaling by inverse layer density. This recipe can match or beat dense validation loss while reducing training computation by up to 25%, including a 3.9B model that achieves validation loss 1.745 with 20% FLOPs savings. The same pretraining induces depth elasticity, allowing early exit and intermediate-layer skipping without retraining; alternating dropout is strongest for non-contiguous layer skipping, while increasing dropout is better for early exit. For post-training Draft & Verify self-speculative decoding on XSUM, the 8.2B model reaches 1.55x speedup, and the 3.9B model reaches 1.54x, where the dense baseline achieves only 1.02x. Overall, layer dropout becomes a unified strategy for cheaper training and flexible inference, implemented on Cerebras CS-3 systems without architectural changes or auxiliary pretraining losses.