InferenceQuantizationMode
Enums · AiDotNet.Configuration
Specifies the weight quantization mode for inference optimization.
For Beginners
These modes control how model weights are compressed for faster inference.
Each mode offers a different trade-off between compression and accuracy:
- INT8: 4x compression, excellent accuracy (per-row scaling)
- FP8: 4x compression, better outlier handling (floating-point format)
- NF4: 8x compression, optimized for normally-distributed weights (QLoRA format)
Fields
| Field | Summary |
|---|---|
InferenceQuantizationMode None |
No weight quantization. |
InferenceQuantizationMode WeightOnlyFP8 |
Per-row FP8 E4M3 weight-only quantization (4x compression). |
InferenceQuantizationMode WeightOnlyInt8 |
Per-row INT8 weight-only quantization (4x compression). |
InferenceQuantizationMode WeightOnlyNF4 |
Per-group NF4 (4-bit NormalFloat) weight-only quantization (8x compression). |
int value__ |