AWQQuantizer<T, TInput, TOutput>

Models & Types · AiDotNet.Deployment.Optimization.Quantization.Strategies

AWQ (Activation-aware Weight Quantization) - protects important weights based on activation magnitudes. Particularly effective for very large models (70B+ parameters).

Implements: IQuantizer<T, TInput, TOutput>

For Beginners

AWQ observes which weights are “activated” most strongly during inference and protects those from aggressive quantization. It’s like knowing which roads are most traveled and keeping those in better condition.

How It Works

How It Works:

Key Features:

Reference: Lin et al., “AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration” (2024)

Constructors

Constructor Summary
AWQQuantizer(QuantizationConfiguration config) Initializes a new instance of the AWQQuantizer.

Properties

Property Summary
QuantizationMode Mode { get; }
bool IsCalibrated { get; } Gets whether the quantizer has been calibrated.
bool UsedRealForwardPasses { get; } Gets whether calibration used real forward passes through the model.
int BitWidth { get; }

Methods

Method Summary
IFullModel<T, TInput, TOutput> Quantize(IFullModel<T, TInput, TOutput> model, QuantizationConfiguration config)
double GetScaleFactor(string layerName)
int GetZeroPoint(string layerName)
void Calibrate(IFullModel<T, TInput, TOutput> model, IEnumerable<TInput> calibrationData)