QuantizationConfiguration

Options & Configuration · AiDotNet.Deployment.Optimization.Quantization

Configuration for model quantization - comprehensive settings for PTQ and QAT.

For Beginners

Quantization compresses your model by using smaller numbers. This configuration lets you control exactly how that compression happens.

How It Works

Quick Start Examples:

Research References:

Constructors

Constructor Summary
QuantizationConfiguration()

Properties

Property Summary
CalibrationMethod CalibrationMethod { get; set; } Gets or sets the calibration method.
Dictionary<LayerCategory, int> CategoryBitWidths { get; set; } Gets or sets per-category bit-width overrides for mixed-precision quantization.
Dictionary<string, LayerQuantizationParams> CustomLayerParams { get; set; } Gets or sets custom quantization parameters per layer (by layer name).
HashSet<string> SkipLayers { get; set; } Gets or sets layers to skip during quantization.
QATMethod QATMethod { get; set; } Gets or sets the QAT method to use when UseQuantizationAwareTraining is true.
QuantizationGranularity Granularity { get; set; } Gets or sets the quantization granularity (where to apply scaling factors).
QuantizationMode Mode { get; set; } Gets or sets the quantization mode (Int8, Float16, etc.).
QuantizationStrategy Strategy { get; set; } Gets or sets the quantization strategy (algorithm) to use.
bool GPTQActOrder { get; set; } Gets or sets whether to use ActOrder optimization in GPTQ.
bool QuantizeActivations { get; set; } Gets or sets whether to quantize only weights or both weights and activations.
bool UsePerChannelQuantization { get; set; } Gets or sets whether to use per-channel quantization.
bool UseQuantizationAwareTraining { get; set; } Gets or sets whether to use quantization-aware training (QAT).
bool UseSymmetricQuantization { get; set; } Gets or sets whether to use symmetric quantization (default: true).
double AWQProtectionPercentage { get; set; } Gets or sets the protection percentage for AWQ strategy.
double GPTQDampingFactor { get; set; } Gets or sets the damping factor for GPTQ Hessian computation.
double HistogramPercentile { get; set; } Gets or sets the percentile to use for histogram-based calibration.
double MaxScaleFactor { get; set; } Gets or sets the maximum scale factor to prevent overflow.
double MinScaleFactor { get; set; } Gets or sets the minimum scale factor to prevent underflow.
double SmoothQuantAlpha { get; set; } Gets or sets the smoothing factor alpha for SmoothQuant strategy.
double[] AWQScaleSearchOptions { get; set; } Gets or sets the scale search options for AWQ grid search optimization.
int ActivationBitWidth { get; set; } Gets or sets the bit width for activation quantization (if QuantizeActivations is true).
int BitWidth { get; } Gets the bit width for the current quantization mode.
int EffectiveBitWidth { get; } Gets the effective bit width (target or default based on mode).
int GroupSize { get; set; } Gets or sets the group size for per-group quantization.
int NumCalibrationSamples { get; set; } Gets or sets the number of calibration samples to use.
int QATWarmupEpochs { get; set; } Gets or sets the number of QAT warmup epochs before enabling fake quantization.
int? TargetBitWidth { get; set; } Gets or sets the target bit width for weight quantization.

Methods

Method Summary
QuantizationConfiguration ForAWQ(int groupSize, double protectionPercentage) Creates a configuration for AWQ 4-bit quantization.
QuantizationConfiguration ForDynamic() Creates a configuration for dynamic quantization (weights only).
QuantizationConfiguration ForFloat16() Creates a configuration for FP16 quantization.
QuantizationConfiguration ForGPTQ(int groupSize, bool actOrder) Creates a configuration for GPTQ 4-bit quantization.
QuantizationConfiguration ForInt8(CalibrationMethod method) Creates a configuration for INT8 quantization.
QuantizationConfiguration ForMixedPrecision(int sensitiveBitWidth, int aggressiveBitWidth, int groupSize) Creates a mixed-precision configuration for layer-aware quantization.
QuantizationConfiguration ForQAT(int targetBitWidth, QATMethod method) Creates a configuration for Quantization-Aware Training (QAT).
QuantizationConfiguration ForQLoRA() Creates a configuration optimized for 4-bit QLoRA fine-tuning.
QuantizationConfiguration ForSmoothQuant(double alpha) Creates a configuration for SmoothQuant W8A8 quantization.
int GetBitWidthForCategory(LayerCategory category) Gets the effective bit-width for a specific layer category, considering per-category overrides in CategoryBitWidths.
int GetBitWidthForLayer(LayerInfo<T> layerInfo) Gets the effective bit-width for a specific layer, checking name-based overrides first, then category-based overrides, then the default.