Huawei's MACaQ takes aim at the cost of quantizing big models

The reason we have large language models on hardware that never should have been able to contain them is because of the process of quantizing them. Converting weights from 16-bit float to 4-bit integers is a way to cut the memory footprint by roughly a factor of 4. It also reduces the bandwidth requirements for accessing the weights (often a bottleneck for inference). But this comes at the expense of accuracy. So the engineering challenge is to minimize the loss of accuracy. Huawei's Computer Systems Lab just released MACaQ, a quantizer that purports to achieve that goal with less time spent calibrating (than existing techniques) and does this for text models, image models, video models and audio models.

But the most interesting claim for local inference isn't the headline accuracy metric. It's that the refinement step (MACaQ-R) keeps the native GGUF format and block scales intact. So you can use the resulting file with a standard installation of llama.cpp. This seems like a trivial detail. But for local use, it's the difference between a research artifact and something that you can download and use on a laptop.

What quantization has to trade

The weights of a model are a bunch of floating point numbers. Quantization is the process of converting that to a smaller list of integers and a scale. At runtime, this is approximated on the fly. The simplest version of this is to have a single scale for all tensors. A more sophisticated approach uses blocks where a small set of weights are grouped with the same scale. This prevents local outliers from requiring the entire tensor to use a coarse step size. This is the basis of the k-quants in GGUF. They use different block sizes and mixed precision for parts of the model that are more sensitive.

More bits -> less error -> more memory. Fewer bits -> less memory -> more error. The challenge with low-bit quantizing is that the error isn't constant. Some weights have large magnitudes and are important. Many weights are near zero and can be aggressively rounded. A quantizer that applies the same treatment to all weights will waste precision on easy cases and hurt the hard cases.

But there's another subtlety that isn't always appreciated. Quantization is often talked about as a way to compress weights. But activations (the intermediate values calculated at runtime) are also part of the inference process. With weight-only quantizing, activations remain at higher precision. This is simpler and can work well for many cases. With weight-and-activation quantizing (W4A4) you have four bits for both. But this is more challenging because the activation ranges depend on the input (and are unknown). MACaQ supports both approaches (depending on the backend).

Why calibration is expensive

Post-training quantizing techniques like GPTQ and AWQ are not just about rounding weights. They use a small calibration dataset to understand what's going on and then make the appropriate adjustments to the quantizing to minimize the loss. GPTQ is about reconstructing layer by layer. AWQ is about activation-aware scaling to protect the most important channels. Both techniques work well. But they require time and memory during the quantizing process.

That's what MACaQ is trying to address. The claim is that it can achieve the same best perplexity for Llama3-8B as MR-GPTQ and DuQuant++ but with eleven times lower time for quantizing. It also claims it can quantize a 122B model in 64 minutes with 32.5 GB of device memory. Note that these numbers are from the vendor. From their own benchmarks. Take them with that in mind. But if they're even approximately right, then they expand the set of people who can quantize a large model. A 122B model that requires a multi-GPU cluster to quantize is not something most people can do. But a 122B model that can fit in 32.5 GB is.

And the other part of the claim is on quality. For Qwen3.6-35B-A3B, MACaQ shows KL divergence to FP16 of 0.013367 for their MACaQ-R Q4_K_XL file. Vs 0.015731 for UD-Q4_K_XL from Unsloth. Both files are the same size: 22.36 GB. KL divergence is a measure of how far the output distribution of the quantized model differs from the full-precision model. So lower is better. It's not a huge difference. But it's in the right direction. And they achieve this without a larger file size.

What MACaQ does differently

It's called Model Agnostic Calibrated Quantization. The "model agnostic" part is important. A lot of quantizing techniques are optimized for a particular architecture or a particular runtime or a particular family of models. MACaQ's calibration is designed to work for different architectures. In the repository they apply it to dense transformers, mixture of experts models, diffusion transformers and speech generators.

The key part is the MACaQ-R step. This refines the codes of the weights in an already-quantized GGUF. But it keeps the native formats and block scales intact. So it doesn't create a new file format. And it doesn't require a patched runtime. It takes a standard quantized file and modifies the codes to reduce error compared to the original float weights. And it writes it back in the same layout. The refinement uses the original model as a teacher. And uses a calibration set to guide the refinement.

This is a practical approach. A new format can be better in the lab. But it may not be practical. No one can use it. By working within the GGUF, MACaQ-R can leverage the whole llama.cpp ecosystem. The same loaders. Same backends. Same tools. So the user can download a file. And run it with llama cli or llama serve. And get the benefit without changing anything else. In the repository, you can see the refinement command with the quantized model as input. And the original Hugging Face model. A teacher GGUF. A calibration text file.

The full MACaQ process with activation transforms is a different story. It uses a bundled runtime. And it goes further than MACaQ-R. That's where the W4A4 results come from. It's important to distinguish. MACaQ-R is the low-friction approach. Works with stock llama.cpp. But full MACaQ is the higher-effort approach. Can quantize activations too.

The backend story

The repository is organized around backends. And each backend is focused on a specific modality. For text generation with 2-/3-/4-bit GGUF files. (Including CPU execution). The llamacpp backend. For text generation and serving. With W4A4 or weight-only modes. The vllm backend. For image, video and audio generation. W4A4. Example models: FLUX.2-klein-4B for images. Wan2.2-TI2V-5B for video. stable-audio-1.0 for audio. The diffusers backend. For speech generation. W4A4. Example models: Qwen3-TTS. Chatterbox Turbo. The pytorch backend.

There is also an Ascend approach. The Huawei CANN stack. They have W4A4 dense and grouped matmul kernels. Through PRs to ops-nn and ops-tilelang. With PTO-ISA and TileLang. Using the CCE backend. To do full model execution on Ascend you still need to integrate with an Ascend inference runtime. So this is for people working on Huawei hardware. Not a turnkey approach for everyone else.

But the point is the breadth. Most of the research on quantizing is focused on text. Because that's where the benchmarks are. And that's where the demand is. The claim for MACaQ is that this same calibration idea can be applied to diffusion transformers. And to speech models. With different activation statistics. And different sensitivity profiles. If this generalizes, then it's a useful primitive for anyone deploying generative models on limited hardware.

The numbers, and what they do not tell you

Impressive numbers in the headline. But it's not the full story. KL divergence is a distribution level metric. It correlates with quality. But it doesn't tell you if the model is better at code. Or worse at math. Or more prone to hallucinate. A quantized model could have low average divergence but not do well on your specific task. So if you're deploying a quantized model then you should do your own evaluations on your own data.

And the speed numbers and memory numbers are benchmark specific. "64 minutes for 122B on 32.5 GB" depends on the hardware. And the size of the calibration set. And the number of bits. For the exact configuration, check the examples in the repository. And the paper. Take the numbers as evidence that the approach is efficient. But don't take them as a guarantee for your specific case.

And as always, take vendor benchmarks with a grain of salt. Huawei's lab did the work. And wrote the paper. And made the comparison. So that doesn't mean the results are wrong. But you need independent replication to go from claim to fact. The fact that you can run MACaQ-R files with stock llama.cpp makes it easy to replicate. So that's a plus for the project.

What to check before adopting

So the first thing to check is if the model you're interested in is in the list of supported models in the repository. Or if it's using one of the recipes. They have examples of dense Qwen3 models. And mixture of experts models. And a coder variant. And diffusion and speech models. But "model agnostic" doesn't mean "every architecture works out of the box". The calibration is assuming something about how the model is processing inputs. So if your model has a weird attention pattern or a custom normalization layer, you might need to tweak it.The second is the calibration data. Quantization quality depends on the data used to calibrate, and a calibration set that does not resemble your workload can produce a model that scores well on average and poorly on your task. The repository ships a calibration text file for its examples; for production use, you should build one from representative inputs and evaluate on a held-out set.

The third is the runtime. MACaQ-R is designed to produce files that run with standard llama.cpp, which is its main practical advantage. The W4A4 paths for vLLM and Diffusers require the matching backend and, in some cases, a bundled runtime. Before committing to a pipeline, confirm that the quantized artifact loads in the runtime you actually deploy and that the kernels for your hardware are available.

The fourth is the evaluation. KL divergence is a useful proxy, but it is not a substitute for task-level testing. Run the quantized model on the prompts and benchmarks that matter to you, compare it with the full-precision model and with the quantized baseline you would otherwise use, and decide whether the difference is acceptable. The repository makes this easy because the files are standard GGUF, but the evaluation is still your responsibility.

Finally, check the license and the provenance. MACaQ is Apache-2.0, which is permissive, but the models you quantize may carry their own licenses. Quantizing a model does not change its license terms, and distributing a quantized derivative can trigger attribution or use restrictions. That is a legal question, not a technical one, and it is worth resolving before a quantized model ships.

Where this fits

The practical audience for MACaQ is people who run models locally or at the edge and who care about the quality of low-bit files. If you already use llama.cpp and GGUF, MACaQ-R is a drop-in refinement step that may improve a quantized model without changing its size. If you serve models with vLLM or build diffusion pipelines, the W4A4 backends are more relevant, though they require more setup.

The strategic audience is the quantization ecosystem itself. GPTQ, AWQ, and the GGUF k-quants are mature and well understood, and a new method has to earn trust. MACaQ's bet is that model-agnostic calibration plus format compatibility is a better combination than a new format with better numbers. That is a sensible bet. Compatibility is what makes a method spread.

Conclusion

MACaQ is a credible entry in the crowded quantization space. Its claims, better low-bit quality at the same file size and far less calibration time, are the right claims for the current moment, when model sizes are growing faster than consumer memory. Its decision to preserve GGUF and work with standard llama.cpp makes it usable rather than merely interesting.

The open questions are the usual ones: independent benchmarks, task-specific quality, and whether the activation-quantization paths are as robust as the weight-only ones. The repository is Apache-2.0 and the paper is public, so those questions can be answered by anyone willing to run the experiments. For a field that often publishes methods nobody can reproduce, that openness is worth more than another point of KL divergence, and it is the main reason to take MACaQ seriously.

Sources

• MACaQ repository and README: https://github.com/huawei-csl/MACaQ
• MACaQ paper, "Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations": https://arxiv.org/abs/2611.10649
• MACaQ blog: https://huawei-csl.com/blog/macaq/
• Qwen3.6-35B-A3B MACaQ-R GGUF: https://huggingface.co/huawei-csl/Qwen3.6-35B-A3B-MACaQ-R-GGUF

← back to posts