FLAMIA: FLAsh MIcroscaling Attention for Fast and Causally Safe Training on Blackwell
AuthorsFedor Velikonivtsev, Mikhail Koltakov, Denis Kuznedelev
AffiliationsYandex Research
FLAMIA makes Transformer attention training faster on Blackwell GPUs by carefully combining low-precision computation with safeguards for causal correctness.
Key results
Upper end of the speedup range versus the fastest baseline on causal packed-document workloads.
Upper end of the speedup range versus the fastest baseline on causal packed-document workloads.
Parameter count of the Llama-3-style model used in pretraining validation.
Token count used to pretrain the Llama-3-style model.
Compared with 2.392 for the BF16 baseline.
What the paper found
FLAMIA is a family of low-precision attention kernels for NVIDIA Blackwell GPUs, designed to turn FP8 and FP4 hardware throughput into faster training without sacrificing causal correctness. Its MXFP8 and mixed NVFP4/FP8 paths use token-block scaling so quantized operands can be reused across matrix orientations, while aligned forward and backward computations reduce training instability. To prevent future tokens from influencing earlier outputs through shared quantization scales, FLAMIA limits most scale dependencies to 32-token blocks and offers exact causal safeguards for supported batched workloads. It also processes packed documents directly, without repacking, and uses GPU-based workload scheduling to balance variable-length sequences. Across causal packed-document benchmarks, its combined forward-and-backward attention call, including quantization, is 1.10–1.40× faster on B200 and 1.06–1.42× faster on B300 than the fastest evaluated baseline, including FlashAttention-4 and cuDNN. In a pretraining test, a 1B-parameter Llama-3-style model trained on 100B FineWeb-Edu tokens had final-average loss of 2.397 with MXFP8, compared with 2.392 for the BF16 baseline, with downstream scores remaining close. The results show that low-bit attention performance depends not just on faster matrix multiplication, but also on causal-safe scaling, numerical consistency, and document-aware execution.
Original abstract
Attention remains a substantial computational cost in Transformer training, particularly at long context lengths. NVIDIA Blackwell GPUs provide high-throughput low-precision matrix multiplication, but exploiting it in attention requires addressing quantization overhead, causal information leakage through shared scales, and numerical inconsistencies between the forward and backward passes. We present FLAMIA, a family of attention kernels for Blackwell with MXFP8 and mixed NVFP4/FP8 training paths. Our design combines token-block scaling that reuses quantized operands across matrix orientations, targeted safeguards against scale-induced leakage, and consistent forward and backward computations. Fused quantization and architecture-specific kernel optimizations reduce overhead, while document-local scaling and workload-aware scheduling support variable-length documents directly in packed layouts without repacking. Across various causal packed-document workloads, the MXFP8 path accelerates the combined forward and backward attention call, including quantization, by factors of $1.10$--$1.40\times$ on B200 and $1.06$--$1.42\times$ on B300 relative to the fastest evaluated baseline for each workload. We validate training stability and quality preservation on pretraining Llama-style models. These results show how jointly designing quantization, training numerics, and document scheduling translates low-bit hardware throughput into practical attention acceleration.
Read the original paperMore in Efficient AI
Browse all 79 papers →Latent Feedback Is a Compute and Data Amplifier
Benhao Huang, Chaitanya Dwivedi, Xuezhe Ma, Alexander Moreno, Hector Liu
Latent feedback may let language models get more value from each training token and FLOP than simply repeating the data.
MACaQ: Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations
Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang, Lukas Cavigelli
MACaQ aims to make large neural networks cheaper to run by quantizing their weights and activations quickly, with little added inference overhead.
Mobile AI Factory: Building Specialized Language Models for Your Mobile Device
Evan Li, Krishna Deshpande, Wen Cheng, Shyamnath Gollakota
Mobile AI Factory automatically builds and tunes a small language model for a specific task and phone, aiming to deliver strong performance with less compute.