NTH

FLAMIA: FLAsh MIcroscaling Attention for Fast and Causally Safe Training on Blackwell

AuthorsFedor Velikonivtsev, Mikhail Koltakov, Denis Kuznedelev

AffiliationsYandex Research

November 12, 2026 2 min read
Watch on YouTube
The one-line take

FLAMIA makes Transformer attention training faster on Blackwell GPUs by carefully combining low-precision computation with safeguards for causal correctness.

Key results

1.40
B200 maximum training-call speedup

Upper end of the speedup range versus the fastest baseline on causal packed-document workloads.

1.42
B300 maximum training-call speedup

Upper end of the speedup range versus the fastest baseline on causal packed-document workloads.

1B
Llama-style model size

Parameter count of the Llama-3-style model used in pretraining validation.

100B
FineWeb-Edu training tokens

Token count used to pretrain the Llama-3-style model.

2.397
MXFP8 final-average training loss

Compared with 2.392 for the BF16 baseline.

What the paper found

FLAMIA is a family of low-precision attention kernels for NVIDIA Blackwell GPUs, designed to turn FP8 and FP4 hardware throughput into faster training without sacrificing causal correctness. Its MXFP8 and mixed NVFP4/FP8 paths use token-block scaling so quantized operands can be reused across matrix orientations, while aligned forward and backward computations reduce training instability. To prevent future tokens from influencing earlier outputs through shared quantization scales, FLAMIA limits most scale dependencies to 32-token blocks and offers exact causal safeguards for supported batched workloads. It also processes packed documents directly, without repacking, and uses GPU-based workload scheduling to balance variable-length sequences. Across causal packed-document benchmarks, its combined forward-and-backward attention call, including quantization, is 1.10–1.40× faster on B200 and 1.06–1.42× faster on B300 than the fastest evaluated baseline, including FlashAttention-4 and cuDNN. In a pretraining test, a 1B-parameter Llama-3-style model trained on 100B FineWeb-Edu tokens had final-average loss of 2.397 with MXFP8, compared with 2.392 for the BF16 baseline, with downstream scores remaining close. The results show that low-bit attention performance depends not just on faster matrix multiplication, but also on causal-safe scaling, numerical consistency, and document-aware execution.

Original abstract

Attention remains a substantial computational cost in Transformer training, particularly at long context lengths. NVIDIA Blackwell GPUs provide high-throughput low-precision matrix multiplication, but exploiting it in attention requires addressing quantization overhead, causal information leakage through shared scales, and numerical inconsistencies between the forward and backward passes. We present FLAMIA, a family of attention kernels for Blackwell with MXFP8 and mixed NVFP4/FP8 training paths. Our design combines token-block scaling that reuses quantized operands across matrix orientations, targeted safeguards against scale-induced leakage, and consistent forward and backward computations. Fused quantization and architecture-specific kernel optimizations reduce overhead, while document-local scaling and workload-aware scheduling support variable-length documents directly in packed layouts without repacking. Across various causal packed-document workloads, the MXFP8 path accelerates the combined forward and backward attention call, including quantization, by factors of $1.10$--$1.40\times$ on B200 and $1.06$--$1.42\times$ on B300 relative to the fastest evaluated baseline for each workload. We validate training stability and quality preservation on pretraining Llama-style models. These results show how jointly designing quantization, training numerics, and document scheduling translates low-bit hardware throughput into practical attention acceleration.

Read the original paper

More in Efficient AI

Browse all 79 papers →
01Efficiency

Latent Feedback Is a Compute and Data Amplifier

Benhao Huang, Chaitanya Dwivedi, Xuezhe Ma, Alexander Moreno, Hector Liu

Latent feedback may let language models get more value from each training token and FLOP than simply repeating the data.

Read analysis