NTH

MACaQ: Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations

AuthorsLorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang, Lukas Cavigelli

AffiliationsComputing Systems Lab, Huawei Technologies, Switzerland

November 18, 2026 2 min read
Watch on YouTube
The one-line take

MACaQ aims to make large neural networks cheaper to run by quantizing their weights and activations quickly, with little added inference overhead.

Key results

11×
Llama-3-8B quantization-time advantage

Matches the best reported perplexity of DuQuant++ and MR-GPTQ with less quantization time.

2.3%
Qwen3-32B decode overhead

Overhead from MACaQ's online activation transforms.

3×
Maximum speedup over FP16

Reported end-to-end inference speedup.

122B
Qwen3.5-122B-A10B model size

Model quantized by MACaQ.

64
Large-model quantization time

Minutes to quantize Qwen3.5-122B-A10B.

32.5GB
Peak GPU memory for quantization

GPU memory used while quantizing Qwen3.5-122B-A10B.

What the paper found

MACaQ is a post-training method for quantizing both weights and activations to MXFP4 W4A4. Rather than propagating basis changes across a network, it processes each linear layer independently: per-channel scaling and a block-Hadamard rotation of size 32 prepare activations and weights for quantization, while the inverse weight-side changes are absorbed into the quantized weights. Its Hessian-aware Flip-Refinement then improves round-to-nearest codes by greedily moving individual FP4 values to adjacent representable levels. This layer-local design supports streaming quantization, with only one layer needing to reside on the GPU, and avoids architecture-specific edits. On Llama-3-8B, MACaQ matches the best reported perplexity of DuQuant++ and MR-GPTQ in 11× less quantization time; on Qwen3-32B, its online transforms add just 2.3% decode overhead, while the implementation reaches up to 3× the FP16 speed. The method also quantizes Qwen3.5-122B-A10B in 64 minutes using 32.5GB of GPU memory, producing WikiText-2 perplexity of 5.14 versus 5.06 for the full-precision model. Applying the same layer-local approach to FLUX.2 Klein 4B further demonstrates that it is not limited to language models.

Original abstract

Weight-activation quantization can reduce the memory and compute cost of neural-network inference, but recent methods often require expensive optimization and architecture-specific propagation of basis transforms across layers. We introduce MACaQ, an MXFP4 W4A4 post-training quantization method that treats every linear layer independently. MACaQ applies a per-channel scale and a block-Hadamard transform online to each layer's activations, while absorbing the corresponding inverse transforms into its weights. Retaining these transforms online avoids cross-layer dependencies and enables a simple streaming implementation of the quantization process, in which only one layer must reside in memory at a time. To improve over round-to-nearest quantization, we introduce Flip-Refinement, a Hessian-aware coordinate-descent procedure that greedily moves individual FP4 codes to adjacent representable values. Despite executing the activation transforms at inference time, MACaQ adds only 2.3% end-to-end overhead over pure MXFP4 on Qwen3-32B in vLLM. MACaQ reaches lower perplexity than DuQuant++ and MR-GPTQ, and already matches their best perplexity on Llama-3-8B in 11 times less quantization time. We further apply MACaQ without architecture-specific modifications to an image diffusion model, illustrating the generality of its layer-local design and show quantization of a 122B parameter model in 64 minutes, using only 32.5GB GPU RAM.

Read the original paper

More in Efficient AI

Browse all 79 papers →
01Efficiency

Latent Feedback Is a Compute and Data Amplifier

Benhao Huang, Chaitanya Dwivedi, Xuezhe Ma, Alexander Moreno, Hector Liu

Latent feedback may let language models get more value from each training token and FLOP than simply repeating the data.

Read analysis