Int4Quantizer<T, TInput, TOutput>
Models & Types · AiDotNet.Deployment.Optimization.Quantization
INT4 (4-bit) quantizer for model optimization — the simple MinMax counterpart to
Int8Quantizer, used for QLoRA-style low-bit base weights.
Implements: IQuantizer<T, TInput, TOutput>
For Beginners
4-bit quantization snaps each weight to one of only 16 levels, so the base model is ~8x smaller than FP32. That loses precision, which is exactly why QLoRA keeps a small full-precision LoRA adapter on top — the adapter absorbs what the 4-bit base can’t represent.
How It Works
Symmetric quantization on the signed 4-bit grid [-8, 7] with scale = absMax / 7. Like
Int8Quantizer this is simulated (fake) quantization: values are
rounded onto the 4-bit grid and stored dequantized in T, so the precision
loss is modeled without changing the parameter container.
Constructors
| Constructor | Summary |
|---|---|
Int4Quantizer() |
Properties
| Property | Summary |
|---|---|
QuantizationMode Mode { get; } |
|
int BitWidth { get; } |
Methods
| Method | Summary |
|---|---|
IFullModel<T, TInput, TOutput> Quantize(IFullModel<T, TInput, TOutput> model, QuantizationConfiguration config) |
|
double GetScaleFactor(string layerName) |
|
int GetZeroPoint(string layerName) |
|
void Calibrate(IFullModel<T, TInput, TOutput> model, IEnumerable<TInput> calibrationData) |