NF4Quantizer<T, TInput, TOutput>

Models & Types · AiDotNet.Deployment.Optimization.Quantization.Formats

NF4 (4-bit NormalFloat) quantizer - optimal for normally distributed weights. Used by QLoRA for efficient 4-bit base model quantization.

Implements: IQuantizer<T, TInput, TOutput>

For Beginners

NF4 is a special 4-bit format where the 16 possible values are chosen to be optimal for weights that follow a normal distribution (bell curve). This makes it perfect for neural network weights, which are typically normally distributed.

How It Works

How It Works:

Key Features:

Reference: Dettmers et al., “QLoRA: Efficient Finetuning of Quantized LLMs” (2023)

Constructors

Constructor Summary
NF4Quantizer(QuantizationConfiguration config, int blockSize) Initializes a new instance of the NF4Quantizer.

Properties

Property Summary
QuantizationMode Mode { get; }
bool IsCalibrated { get; set; } Gets whether the quantizer has been calibrated.
int BitWidth { get; }

Methods

Method Summary
IFullModel<T, TInput, TOutput> Quantize(IFullModel<T, TInput, TOutput> model, QuantizationConfiguration config)
double GetScaleFactor(string layerName)
double IndexToNF4(int index) Converts a 4-bit index to its NF4 value.
int GetZeroPoint(string layerName)
int NF4ToIndex(double value) Converts a value to its nearest 4-bit index.
void Calibrate(IFullModel<T, TInput, TOutput> model, IEnumerable<TInput> calibrationData)