SmoothQuantQuantizer<T, TInput, TOutput>

Models & Types · AiDotNet.Deployment.Optimization.Quantization.Strategies

SmoothQuant - transfers quantization difficulty from activations to weights using per-channel smoothing. Enables efficient W8A8 quantization (both weights and activations at 8-bit).

Implements: IQuantizer<T, TInput, TOutput>

For Beginners

Activations (intermediate values during inference) often have outliers that are very hard to quantize. SmoothQuant “smooths” these outliers by mathematically transferring some of their range to the weights, making both easier to quantize.

How It Works

How It Works:

Key Features:

Reference: Xiao et al., “SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models” (2023)

Constructors

Constructor Summary
SmoothQuantQuantizer(QuantizationConfiguration config) Initializes a new instance of the SmoothQuantQuantizer.

Properties

Property Summary
IReadOnlyDictionary<string, double[]> SmoothingScales { get; } Gets the smoothing scales applied to transform weights.
QuantizationMode Mode { get; }
bool IsCalibrated { get; } Gets whether the quantizer has been calibrated.
bool UsedRealForwardPasses { get; } Gets whether calibration used real forward passes through the model.
int BitWidth { get; }

Methods

Method Summary
IFullModel<T, TInput, TOutput> Quantize(IFullModel<T, TInput, TOutput> model, QuantizationConfiguration config)
double GetScaleFactor(string layerName)
int GetZeroPoint(string layerName)
void Calibrate(IFullModel<T, TInput, TOutput> model, IEnumerable<TInput> calibrationData)