GPTQQuantizer<T, TInput, TOutput>
Models & Types · AiDotNet.Deployment.Optimization.Quantization.Strategies
GPTQ (Generative Pre-trained Transformer Quantization) - state-of-the-art weight quantization using second-order Hessian information to minimize reconstruction error.
Implements: IQuantizer<T, TInput, TOutput>
For Beginners
GPTQ is like a smart packing algorithm that knows which items are most important. It uses advanced math (Hessian matrix) to figure out which weights matter most and handles those more carefully during compression.
How It Works
How It Works:
Key Features:
Reference: Frantar et al., “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers” (2023)
Constructors
| Constructor | Summary |
|---|---|
GPTQQuantizer(QuantizationConfiguration config) |
Initializes a new instance of the GPTQQuantizer. |
Properties
| Property | Summary |
|---|---|
QuantizationMode Mode { get; } |
|
bool IsCalibrated { get; } |
Gets whether the quantizer has been calibrated. |
bool UsedRealForwardPasses { get; } |
Gets whether calibration used real forward passes through the model. |
int BitWidth { get; } |
Methods
| Method | Summary |
|---|---|
IFullModel<T, TInput, TOutput> Quantize(IFullModel<T, TInput, TOutput> model, QuantizationConfiguration config) |
|
double GetScaleFactor(string layerName) |
|
int GetZeroPoint(string layerName) |
|
void Calibrate(IFullModel<T, TInput, TOutput> model, IEnumerable<TInput> calibrationData) |