AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
Abstract
Large language models (LLMs) have transformed numerous AI applications. On-device LLM is becoming increasingly important: running LLMs locally on edge devices can reduce the cloud computing cost and protect users' privacy. However, the astronomical model size and the limited hardware resource pose significant deployment challenges. We propose Activation-aware Weight Quantization (AWQ), a hardware-friendly approach for LLM low-bit weight-only quantization. AWQ finds that not all weights in an LLM are equally important. Protecting only 1% salient weights can greatly reduce quantization error. To identify salient weight channels, we should refer to the activation distribution, not weights. To avoid the hardware-inefficient mix-precision quantization, we mathematically derive that scaling up the salient channels can reduce the quantization error. AWQ employs an equivalent transformation to scale the salient weight channels to protect them. The scale is determined by collecting the activation statistics offline. AWQ does not rely on any backpropagation or reconstruction, so it generalizes to different domains and modalities without overfitting the calibration set. AWQ outperforms existing work on various language modeling and domain-specific benchmarks (coding and math). Thanks to better generalization, it achieves excellent quantization performance for instruction-tuned LMs and, for the first time, multi-modal LMs. Alongside AWQ, we implement TinyChat, an efficient and flexible inference framework tailored for 4-bit on-device LLM/VLMs. With kernel fusion and platform-aware weight packing, TinyChat offers more than 3x speedup over the Huggingface FP16 implementation on both desktop and mobile GPUs. It also democratizes the deployment of the 70B Llama-2 model on mobile GPUs.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of AWQ, a novel quantization method that significantly reduces quantization error by focusing on activation distributions to identify salient weights, enabling efficient deployment of large language models on edge devices. This work is poised to influence future research in model compression and acceleration, particularly in resource-constrained environments.
Methodology
The proposed Activation-aware Weight Quantization (AWQ) method presents a novel approach by focusing on the activation distribution rather than weights to identify salient weight channels. This is a significant shift in perspective that could lead to more efficient quantization strategies. The mathematical derivation for scaling salient channels is well-founded, and the method's independence from backpropagation enhances its applicability across different domains and modalities. However, the paper could benefit from a more detailed explanation of the mathematical transformations and their implications.
Results & evaluation
The experiments conducted demonstrate AWQ's superiority over existing quantization methods across various benchmarks, including language modeling and domain-specific tasks. The reported performance improvements, particularly in terms of quantization error reduction and inference speed, are compelling. However, the paper lacks detailed descriptions of the datasets used, which would help in assessing the generalizability of the results.
Reproducibility
The paper does not provide specific implementation details or code repositories, which raises concerns about reproducibility. While the methodology is described, the absence of a public implementation limits the ability of other researchers to verify the results and apply the method to their own tasks.
Limitations
One limitation is the reliance on offline activation statistics, which may not be feasible in all deployment scenarios. Additionally, while the method claims to generalize well, the paper does not provide extensive testing across a wide range of models or tasks, which could limit its applicability.
Broader impact
The ability to run large language models on edge devices has significant implications for privacy and cost reduction in AI applications. AWQ could democratize access to advanced AI capabilities on mobile devices, potentially leading to broader adoption of LLMs in various fields. The main contribution of this paper is the introduction of AWQ, a novel quantization method that significantly reduces quantization error by focusing on activation distributions to identify salient weights, enabling efficient deployment of large language models on edge devices. This work is poised to influence future research in model compression and acceleration, particularly in resource-constrained environments.
Curator note
Lin et al.; better quantization by protecting salient weights
Author of this paper? Send a correction.