AWQQuantizer<T, TInput, TOutput>
Models & Types · AiDotNet.Deployment.Optimization.Quantization.Strategies
AWQ (Activation-aware Weight Quantization) - protects important weights based on activation magnitudes. Particularly effective for very large models (70B+ parameters).
Implements: IQuantizer<T, TInput, TOutput>
For Beginners
AWQ observes which weights are “activated” most strongly during inference and protects those from aggressive quantization. It’s like knowing which roads are most traveled and keeping those in better condition.
How It Works
How It Works:
Key Features:
Reference: Lin et al., “AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration” (2024)
Constructors
| Constructor | Summary |
|---|---|
AWQQuantizer(QuantizationConfiguration config) |
Initializes a new instance of the AWQQuantizer. |
Properties
| Property | Summary |
|---|---|
QuantizationMode Mode { get; } |
|
bool IsCalibrated { get; } |
Gets whether the quantizer has been calibrated. |
bool UsedRealForwardPasses { get; } |
Gets whether calibration used real forward passes through the model. |
int BitWidth { get; } |
Methods
| Method | Summary |
|---|---|
IFullModel<T, TInput, TOutput> Quantize(IFullModel<T, TInput, TOutput> model, QuantizationConfiguration config) |
|
double GetScaleFactor(string layerName) |
|
int GetZeroPoint(string layerName) |
|
void Calibrate(IFullModel<T, TInput, TOutput> model, IEnumerable<TInput> calibrationData) |