MACaQ: Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations
AuthorsLorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang, Lukas Cavigelli
AffiliationsComputing Systems Lab, Huawei Technologies, Switzerland
MACaQ aims to make large neural networks cheaper to run by quantizing their weights and activations quickly, with little added inference overhead.
Key results
Matches the best reported perplexity of DuQuant++ and MR-GPTQ with less quantization time.
Overhead from MACaQ's online activation transforms.
Reported end-to-end inference speedup.
Model quantized by MACaQ.
Minutes to quantize Qwen3.5-122B-A10B.
GPU memory used while quantizing Qwen3.5-122B-A10B.
What the paper found
MACaQ is a post-training method for quantizing both weights and activations to MXFP4 W4A4. Rather than propagating basis changes across a network, it processes each linear layer independently: per-channel scaling and a block-Hadamard rotation of size 32 prepare activations and weights for quantization, while the inverse weight-side changes are absorbed into the quantized weights. Its Hessian-aware Flip-Refinement then improves round-to-nearest codes by greedily moving individual FP4 values to adjacent representable levels. This layer-local design supports streaming quantization, with only one layer needing to reside on the GPU, and avoids architecture-specific edits. On Llama-3-8B, MACaQ matches the best reported perplexity of DuQuant++ and MR-GPTQ in 11× less quantization time; on Qwen3-32B, its online transforms add just 2.3% decode overhead, while the implementation reaches up to 3× the FP16 speed. The method also quantizes Qwen3.5-122B-A10B in 64 minutes using 32.5GB of GPU memory, producing WikiText-2 perplexity of 5.14 versus 5.06 for the full-precision model. Applying the same layer-local approach to FLUX.2 Klein 4B further demonstrates that it is not limited to language models.
Original abstract
Weight-activation quantization can reduce the memory and compute cost of neural-network inference, but recent methods often require expensive optimization and architecture-specific propagation of basis transforms across layers. We introduce MACaQ, an MXFP4 W4A4 post-training quantization method that treats every linear layer independently. MACaQ applies a per-channel scale and a block-Hadamard transform online to each layer's activations, while absorbing the corresponding inverse transforms into its weights. Retaining these transforms online avoids cross-layer dependencies and enables a simple streaming implementation of the quantization process, in which only one layer must reside in memory at a time. To improve over round-to-nearest quantization, we introduce Flip-Refinement, a Hessian-aware coordinate-descent procedure that greedily moves individual FP4 codes to adjacent representable values. Despite executing the activation transforms at inference time, MACaQ adds only 2.3% end-to-end overhead over pure MXFP4 on Qwen3-32B in vLLM. MACaQ reaches lower perplexity than DuQuant++ and MR-GPTQ, and already matches their best perplexity on Llama-3-8B in 11 times less quantization time. We further apply MACaQ without architecture-specific modifications to an image diffusion model, illustrating the generality of its layer-local design and show quantization of a 122B parameter model in 64 minutes, using only 32.5GB GPU RAM.
Read the original paperMore in Efficient AI
Browse all 79 papers →Latent Feedback Is a Compute and Data Amplifier
Benhao Huang, Chaitanya Dwivedi, Xuezhe Ma, Alexander Moreno, Hector Liu
Latent feedback may let language models get more value from each training token and FLOP than simply repeating the data.
Mobile AI Factory: Building Specialized Language Models for Your Mobile Device
Evan Li, Krishna Deshpande, Wen Cheng, Shyamnath Gollakota
Mobile AI Factory automatically builds and tunes a small language model for a specific task and phone, aiming to deliver strong performance with less compute.
FLAMIA: FLAsh MIcroscaling Attention for Fast and Causally Safe Training on Blackwell
Fedor Velikonivtsev, Mikhail Koltakov, Denis Kuznedelev
FLAMIA makes Transformer attention training faster on Blackwell GPUs by carefully combining low-precision computation with safeguards for causal correctness.