Int4Quantizer<T, TInput, TOutput>

Models & Types · AiDotNet.Deployment.Optimization.Quantization

INT4 (4-bit) quantizer for model optimization — the simple MinMax counterpart to Int8Quantizer, used for QLoRA-style low-bit base weights.

Implements: IQuantizer<T, TInput, TOutput>

For Beginners

4-bit quantization snaps each weight to one of only 16 levels, so the base model is ~8x smaller than FP32. That loses precision, which is exactly why QLoRA keeps a small full-precision LoRA adapter on top — the adapter absorbs what the 4-bit base can’t represent.

How It Works

Symmetric quantization on the signed 4-bit grid [-8, 7] with scale = absMax / 7. Like Int8Quantizer this is simulated (fake) quantization: values are rounded onto the 4-bit grid and stored dequantized in T, so the precision loss is modeled without changing the parameter container.

Constructors

Constructor Summary
Int4Quantizer()

Properties

Property Summary
QuantizationMode Mode { get; }
int BitWidth { get; }

Methods

Method Summary
IFullModel<T, TInput, TOutput> Quantize(IFullModel<T, TInput, TOutput> model, QuantizationConfiguration config)
double GetScaleFactor(string layerName)
int GetZeroPoint(string layerName)
void Calibrate(IFullModel<T, TInput, TOutput> model, IEnumerable<TInput> calibrationData)