QuantizationConfig

Options & Configuration ยท AiDotNet.Deployment.Configuration

Configuration for model quantization - compressing models by using lower precision numbers.

For Beginners

Quantization makes your AI model smaller and faster by using smaller numbers. Think of it like compressing a high-quality photo - it takes less space but might lose a little quality.

How It Works

Why use quantization?

Quick Start Examples:

Constructors

Constructor Summary
QuantizationConfig()

Properties

Property Summary
CalibrationMethod CalibrationMethod { get; set; } Gets or sets the calibration method used to determine optimal scaling factors (default: MinMax).
QATMethod QATMethod { get; set; } Gets or sets the QAT method to use when UseQuantizationAwareTraining is true.
QuantizationGranularity Granularity { get; set; } Gets or sets the quantization granularity (where to apply scaling factors).
QuantizationMode Mode { get; set; } Gets or sets the quantization mode (default: None).
QuantizationStrategy Strategy { get; set; } Gets or sets the quantization strategy (algorithm) to use.
bool QuantizeActivations { get; set; } Gets or sets whether to quantize only weights or both weights and activations (default: false).
bool UseQuantizationAwareTraining { get; set; } Gets or sets whether to use Quantization-Aware Training (QAT) instead of Post-Training Quantization.
bool UseSymmetricQuantization { get; set; } Gets or sets whether to use symmetric quantization (default: true).
int CalibrationSamples { get; set; } Gets or sets the number of calibration samples to use (default: 100).
int GroupSize { get; set; } Gets or sets the group size for per-group quantization (default: 128).
int QATWarmupEpochs { get; set; } Gets or sets the number of warmup epochs before enabling quantization in QAT.
int? TargetBitWidth { get; set; } Gets or sets the target bit width for weight quantization.

Methods

Method Summary
QuantizationConfig ForAWQ(int groupSize) Creates a configuration for AWQ 4-bit quantization.
QuantizationConfig ForGPTQ(int groupSize) Creates a configuration for GPTQ 4-bit quantization.
QuantizationConfig ForInt4() Creates a configuration for simple 4-bit (INT4) weight quantization via MinMax โ€” the low-bit base used by QLoRA (~8x smaller than FP32).
QuantizationConfig ForQAT(int targetBitWidth, QATMethod method) Creates a configuration for Quantization-Aware Training (QAT).
QuantizationConfig ForSmoothQuant() Creates a configuration for SmoothQuant W8A8 quantization.