GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
Abstract
Generative Pre-trained Transformer models, known as GPT or OPT, set themselves apart through breakthrough performance across complex language modelling tasks, but also by their extremely high computational and storage costs. Specifically, due to their massive size, even inference for large, highly-accurate GPT models may require multiple performant GPUs, which limits the usability of such models. While there is emerging work on relieving this pressure via model compression, the applicability and performance of existing compression techniques is limited by the scale and complexity of GPT models. In this paper, we address this challenge, and propose GPTQ, a new one-shot weight quantization method based on approximate second-order information, that is both highly-accurate and highly-efficient. Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline. Our method more than doubles the compression gains relative to previously-proposed one-shot quantization methods, preserving accuracy, allowing us for the first time to execute an 175 billion-parameter model inside a single GPU for generative inference. Moreover, we also show that our method can still provide reasonable accuracy in the extreme quantization regime, in which weights are quantized to 2-bit or even ternary quantization levels. We show experimentally that these improvements can be leveraged for end-to-end inference speedups over FP16, of around 3.25x when using high-end GPUs (NVIDIA A100) and 4.5x when using more cost-effective ones (NVIDIA A6000). The implementation is available at https://github.com/IST-DASLab/gptq.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
GPTQ introduces a highly efficient, second-order post-training quantization algorithm that enables 3-4 bit compression of 100B+ parameter LLMs with negligible accuracy loss, fundamentally lowering the hardware barrier for LLM deployment through clever algorithmic simplifications and hardware-aware optimizations. The paper's combination of theoretical grounding (Hessian-based error compensation), practical systems engineering (lazy batching, Cholesky stability, custom kernels), and extensive empirical validation across the largest publicly available models represents a landmark contribution to efficient ML, directly enabling the widespread adoption of quantized LLMs in both academic and industrial settings.
Methodology
The paper presents a highly effective algorithmic and systems-level adaptation of Optimal Brain Quantization (OBQ), a second-order post-training quantization method, to the scale of modern LLMs. The core methodological breakthrough lies in three synergistic optimizations: (1) demonstrating that arbitrary column ordering yields near-identical accuracy to greedy weight selection, enabling shared Hessian inverse updates across all rows and reducing complexity from O(d_row * d_col^3) to O(max(d_row, d_col) * d_col^2); (2) introducing lazy batch-updates (processing 128-column blocks) to overcome GPU memory bandwidth bottlenecks and maximize compute utilization; and (3) reformulating the iterative inverse updates via a numerically stable Cholesky decomposition, preventing matrix indefiniteness at scale. This represents a masterclass in algorithmic simplification paired with hardware-aware engineering, transforming a theoretically sound but computationally prohibitive technique into a practical, scalable pipeline.
Results & evaluation
The experimental design is rigorous, comprehensive, and highly convincing. The authors evaluate across the full OPT and BLOOM model families (up to 175B/176B parameters), using standard perplexity benchmarks (WikiText2, PTB, C4) and zero-shot reasoning tasks (LAMBADA, ARC, PIQA). Results clearly establish that GPTQ maintains near-FP16 perplexity at 3-4 bits, drastically outperforming Round-to-Nearest (RTN) baselines which collapse at 3-bit. Runtime benchmarks demonstrate ~4-hour quantization on a single A100, and custom CUDA dequantization kernels yield 3.25x–4.5x end-to-end inference speedups over FP16. Additional ablations on grouping granularity and extreme 2-bit/ternary quantization further validate the method's robustness and flexibility.
Reproducibility
Excellent. The authors provide a fully open-source PyTorch implementation with optimized CUDA kernels, calibration scripts, and evaluation harnesses. The methodology is meticulously documented, including the use of 128 random C4 segments for calibration, per-row asymmetric quantization, and a block-wise memory management strategy that enables quantization on hardware with less VRAM than the full model requires. This level of transparency and practical engineering consideration ensures straightforward replication and immediate integration into downstream frameworks.
Limitations
The authors correctly acknowledge that GPTQ does not reduce FLOP counts for matrix multiplications due to the lack of native hardware support for mixed-precision operations (e.g., FP16 × INT4) on mainstream GPUs at the time of publication. The method focuses exclusively on weight quantization, omitting activation quantization, which limits its efficiency in compute-bound, large-batch regimes. Furthermore, the evaluation relies primarily on perplexity and standard zero-shot metrics, without investigating downstream impacts on model bias, safety alignment, or complex reasoning capabilities.
Broader impact
GPTQ has fundamentally democratized access to large language models by enabling 3-4 bit compression of 100B+ parameter networks on single consumer/professional GPUs with negligible accuracy degradation. Its algorithmic insights and open-source release have been rapidly adopted and integrated into major inference ecosystems (e.g., Hugging Face Transformers, vLLM, AutoGPTQ), establishing a new industry standard for post-training quantization. While this dramatically accelerates research and deployment, it also amplifies the need for rigorous safety, bias, and alignment evaluations, as highly capable models become accessible to a much broader, less regulated user base. GPTQ introduces a highly efficient, second-order post-training quantization algorithm that enables 3-4 bit compression of 100B+ parameter LLMs with negligible accuracy loss, fundamentally lowering the hardware barrier for LLM deployment through clever algorithmic simplifications and hardware-aware optimizations. The paper's combination of theoretical grounding (Hessian-based error compensation), practical systems engineering (lazy batching, Cholesky stability, custom kernels), and extensive empirical validation across the largest publicly available models represents a landmark contribution to efficient ML, directly enabling the widespread adoption of quantized LLMs in both academic and industrial settings.
Curator note
Frantar et al.; 3/4-bit quantization with minimal quality loss; widely used
Author of this paper? Send a correction.