SmoothQuantQuantizer<T, TInput, TOutput>
Models & Types · AiDotNet.Deployment.Optimization.Quantization.Strategies
SmoothQuant - transfers quantization difficulty from activations to weights using per-channel smoothing. Enables efficient W8A8 quantization (both weights and activations at 8-bit).
Implements: IQuantizer<T, TInput, TOutput>
For Beginners
Activations (intermediate values during inference) often have outliers that are very hard to quantize. SmoothQuant “smooths” these outliers by mathematically transferring some of their range to the weights, making both easier to quantize.
How It Works
How It Works:
Key Features:
Reference: Xiao et al., “SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models” (2023)
Constructors
| Constructor | Summary |
|---|---|
SmoothQuantQuantizer(QuantizationConfiguration config) |
Initializes a new instance of the SmoothQuantQuantizer. |
Properties
| Property | Summary |
|---|---|
IReadOnlyDictionary<string, double[]> SmoothingScales { get; } |
Gets the smoothing scales applied to transform weights. |
QuantizationMode Mode { get; } |
|
bool IsCalibrated { get; } |
Gets whether the quantizer has been calibrated. |
bool UsedRealForwardPasses { get; } |
Gets whether calibration used real forward passes through the model. |
int BitWidth { get; } |
Methods
| Method | Summary |
|---|---|
IFullModel<T, TInput, TOutput> Quantize(IFullModel<T, TInput, TOutput> model, QuantizationConfiguration config) |
|
double GetScaleFactor(string layerName) |
|
int GetZeroPoint(string layerName) |
|
void Calibrate(IFullModel<T, TInput, TOutput> model, IEnumerable<TInput> calibrationData) |