QuantizationConfig
Options & Configuration ยท AiDotNet.Deployment.Configuration
Configuration for model quantization - compressing models by using lower precision numbers.
For Beginners
Quantization makes your AI model smaller and faster by using smaller numbers. Think of it like compressing a high-quality photo - it takes less space but might lose a little quality.
How It Works
Why use quantization?
Quick Start Examples:
Constructors
| Constructor | Summary |
|---|---|
QuantizationConfig() |
Properties
| Property | Summary |
|---|---|
CalibrationMethod CalibrationMethod { get; set; } |
Gets or sets the calibration method used to determine optimal scaling factors (default: MinMax). |
QATMethod QATMethod { get; set; } |
Gets or sets the QAT method to use when UseQuantizationAwareTraining is true. |
QuantizationGranularity Granularity { get; set; } |
Gets or sets the quantization granularity (where to apply scaling factors). |
QuantizationMode Mode { get; set; } |
Gets or sets the quantization mode (default: None). |
QuantizationStrategy Strategy { get; set; } |
Gets or sets the quantization strategy (algorithm) to use. |
bool QuantizeActivations { get; set; } |
Gets or sets whether to quantize only weights or both weights and activations (default: false). |
bool UseQuantizationAwareTraining { get; set; } |
Gets or sets whether to use Quantization-Aware Training (QAT) instead of Post-Training Quantization. |
bool UseSymmetricQuantization { get; set; } |
Gets or sets whether to use symmetric quantization (default: true). |
int CalibrationSamples { get; set; } |
Gets or sets the number of calibration samples to use (default: 100). |
int GroupSize { get; set; } |
Gets or sets the group size for per-group quantization (default: 128). |
int QATWarmupEpochs { get; set; } |
Gets or sets the number of warmup epochs before enabling quantization in QAT. |
int? TargetBitWidth { get; set; } |
Gets or sets the target bit width for weight quantization. |
Methods
| Method | Summary |
|---|---|
QuantizationConfig ForAWQ(int groupSize) |
Creates a configuration for AWQ 4-bit quantization. |
QuantizationConfig ForGPTQ(int groupSize) |
Creates a configuration for GPTQ 4-bit quantization. |
QuantizationConfig ForInt4() |
Creates a configuration for simple 4-bit (INT4) weight quantization via MinMax โ the low-bit base used by QLoRA (~8x smaller than FP32). |
QuantizationConfig ForQAT(int targetBitWidth, QATMethod method) |
Creates a configuration for Quantization-Aware Training (QAT). |
QuantizationConfig ForSmoothQuant() |
Creates a configuration for SmoothQuant W8A8 quantization. |