GPTQQuantizer<T, TInput, TOutput>

Models & Types · AiDotNet.Deployment.Optimization.Quantization.Strategies

GPTQ (Generative Pre-trained Transformer Quantization) - state-of-the-art weight quantization using second-order Hessian information to minimize reconstruction error.

Implements: IQuantizer<T, TInput, TOutput>

For Beginners

GPTQ is like a smart packing algorithm that knows which items are most important. It uses advanced math (Hessian matrix) to figure out which weights matter most and handles those more carefully during compression.

How It Works

How It Works:

Key Features:

Reference: Frantar et al., “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers” (2023)

Constructors

Constructor Summary
GPTQQuantizer(QuantizationConfiguration config) Initializes a new instance of the GPTQQuantizer.

Properties

Property Summary
QuantizationMode Mode { get; }
bool IsCalibrated { get; } Gets whether the quantizer has been calibrated.
bool UsedRealForwardPasses { get; } Gets whether calibration used real forward passes through the model.
int BitWidth { get; }

Methods

Method Summary
IFullModel<T, TInput, TOutput> Quantize(IFullModel<T, TInput, TOutput> model, QuantizationConfiguration config)
double GetScaleFactor(string layerName)
int GetZeroPoint(string layerName)
void Calibrate(IFullModel<T, TInput, TOutput> model, IEnumerable<TInput> calibrationData)