FP8Quantizer<T, TInput, TOutput>

Models & Types · AiDotNet.Deployment.Optimization.Quantization.Formats

FP8 (8-bit Floating Point) quantizer supporting E4M3 and E5M2 formats. Provides better outlier handling than INT8 while maintaining 8-bit efficiency.

Implements: IQuantizer<T, TInput, TOutput>

For Beginners

FP8 is a newer 8-bit format that uses floating-point representation instead of integers. It’s better at handling outliers (extreme values) and requires less calibration than INT8.

How It Works

FP8 Formats:

Key Features:

Reference: NVIDIA FP8 specification and various hardware vendor implementations

Constructors

Constructor Summary
FP8Quantizer(FP8Format format, QuantizationConfiguration config) Initializes a new instance of the FP8Quantizer.

Properties

Property Summary
FP8Format Format { get; } Gets the FP8 format being used (E4M3 or E5M2).
QuantizationMode Mode { get; }
int BitWidth { get; }

Methods

Method Summary
IFullModel<T, TInput, TOutput> Quantize(IFullModel<T, TInput, TOutput> model, QuantizationConfiguration config)
byte E4M3ToByte(double value) Converts an E4M3 double value to byte.
byte E5M2ToByte(double value) Converts an E5M2 double value to byte.
double ByteToE4M3(byte b) Converts a byte to E4M3 double value.
double ByteToE5M2(byte b) Converts a byte to E5M2 double value.
double GetScaleFactor(string layerName)
int GetZeroPoint(string layerName)
void Calibrate(IFullModel<T, TInput, TOutput> model, IEnumerable<TInput> calibrationData)