FP8Quantizer<T, TInput, TOutput>
Models & Types · AiDotNet.Deployment.Optimization.Quantization.Formats
FP8 (8-bit Floating Point) quantizer supporting E4M3 and E5M2 formats. Provides better outlier handling than INT8 while maintaining 8-bit efficiency.
Implements: IQuantizer<T, TInput, TOutput>
For Beginners
FP8 is a newer 8-bit format that uses floating-point representation instead of integers. It’s better at handling outliers (extreme values) and requires less calibration than INT8.
How It Works
FP8 Formats:
Key Features:
Reference: NVIDIA FP8 specification and various hardware vendor implementations
Constructors
| Constructor | Summary |
|---|---|
FP8Quantizer(FP8Format format, QuantizationConfiguration config) |
Initializes a new instance of the FP8Quantizer. |
Properties
| Property | Summary |
|---|---|
FP8Format Format { get; } |
Gets the FP8 format being used (E4M3 or E5M2). |
QuantizationMode Mode { get; } |
|
int BitWidth { get; } |
Methods
| Method | Summary |
|---|---|
IFullModel<T, TInput, TOutput> Quantize(IFullModel<T, TInput, TOutput> model, QuantizationConfiguration config) |
|
byte E4M3ToByte(double value) |
Converts an E4M3 double value to byte. |
byte E5M2ToByte(double value) |
Converts an E5M2 double value to byte. |
double ByteToE4M3(byte b) |
Converts a byte to E4M3 double value. |
double ByteToE5M2(byte b) |
Converts a byte to E5M2 double value. |
double GetScaleFactor(string layerName) |
|
int GetZeroPoint(string layerName) |
|
void Calibrate(IFullModel<T, TInput, TOutput> model, IEnumerable<TInput> calibrationData) |