NF4Quantizer<T, TInput, TOutput>
Models & Types · AiDotNet.Deployment.Optimization.Quantization.Formats
NF4 (4-bit NormalFloat) quantizer - optimal for normally distributed weights. Used by QLoRA for efficient 4-bit base model quantization.
Implements: IQuantizer<T, TInput, TOutput>
For Beginners
NF4 is a special 4-bit format where the 16 possible values are chosen to be optimal for weights that follow a normal distribution (bell curve). This makes it perfect for neural network weights, which are typically normally distributed.
How It Works
How It Works:
Key Features:
Reference: Dettmers et al., “QLoRA: Efficient Finetuning of Quantized LLMs” (2023)
Constructors
| Constructor | Summary |
|---|---|
NF4Quantizer(QuantizationConfiguration config, int blockSize) |
Initializes a new instance of the NF4Quantizer. |
Properties
| Property | Summary |
|---|---|
QuantizationMode Mode { get; } |
|
bool IsCalibrated { get; set; } |
Gets whether the quantizer has been calibrated. |
int BitWidth { get; } |
Methods
| Method | Summary |
|---|---|
IFullModel<T, TInput, TOutput> Quantize(IFullModel<T, TInput, TOutput> model, QuantizationConfiguration config) |
|
double GetScaleFactor(string layerName) |
|
double IndexToNF4(int index) |
Converts a 4-bit index to its NF4 value. |
int GetZeroPoint(string layerName) |
|
int NF4ToIndex(double value) |
Converts a value to its nearest 4-bit index. |
void Calibrate(IFullModel<T, TInput, TOutput> model, IEnumerable<TInput> calibrationData) |