MixedPrecisionConfig
Options & Configuration · AiDotNet.MixedPrecision
Configuration settings for mixed-precision training.
For Beginners
This class contains all the settings you can adjust for mixed-precision training. The default values work well for most models, but you can customize them based on your specific needs.
Key concepts:
- Loss Scaling: Prevents small gradients from becoming zero in FP16
- Dynamic Scaling: Automatically adjusts the loss scale during training
- Master Weights: FP32 copy of parameters for precise updates
- Working Weights: FP16 copy used for forward/backward passes
- FP8 (New): 8-bit formats for 2x throughput on H100+ GPUs
Constructors
| Constructor | Summary |
|---|---|
MixedPrecisionConfig() |
Creates a configuration with default recommended settings. |
Properties
| Property | Summary |
|---|---|
List<string> FP8ExcludedLayers { get; set; } |
Layers to keep in higher precision (FP16/BF16) even when using FP8. |
MixedPrecisionType FP8BackwardFormat { get; set; } |
Format to use for backward pass in FP8 mode (default: E5M2). |
MixedPrecisionType FP8ForwardFormat { get; set; } |
Format to use for forward pass in FP8 mode (default: E4M3). |
MixedPrecisionType PrecisionType { get; set; } |
The precision type to use for mixed-precision training (default: FP16). |
bool EnableDynamicScaling { get; set; } |
Enable dynamic loss scaling (default: true). |
bool FP8PerTensorScaling { get; set; } |
Whether to use per-tensor scaling for FP8 (default: true). |
bool Fp32BatchNorm { get; set; } |
Whether to keep batch normalization layers in FP32 (default: true). |
bool Fp32GradientAccumulation { get; set; } |
Whether to accumulate gradients in FP32 (default: true). |
bool Fp32Loss { get; set; } |
Whether to keep loss computation in FP32 (default: true). |
double InitialLossScale { get; set; } |
Initial loss scale factor (default: 65536 = 2^16). |
double MaxLossScale { get; set; } |
Maximum allowed loss scale (default: 16777216 = 2^24). |
double MinLossScale { get; set; } |
Minimum allowed loss scale (default: 1.0). |
double ScaleBackoffFactor { get; set; } |
Factor by which to multiply scale when decreasing (default: 0.5). |
double ScaleGrowthFactor { get; set; } |
Factor by which to multiply scale when increasing (default: 2.0). |
int ScaleGrowthInterval { get; set; } |
Number of consecutive successful updates before increasing scale (default: 2000). |
Methods
| Method | Summary |
|---|---|
MixedPrecisionConfig Aggressive() |
Creates an aggressive configuration optimized for maximum throughput. |
MixedPrecisionConfig Conservative() |
Creates a conservative configuration optimized for training stability. |
MixedPrecisionConfig ForBF16() |
Creates a configuration for BF16 precision (Ampere+ GPUs). |
MixedPrecisionConfig ForFP8() |
Creates a configuration for FP8 hybrid mode (H100+ GPUs). |
MixedPrecisionConfig ForFP8Transformers() |
Creates a configuration for FP8 training on transformer models. |
string ToString() |
Gets a summary of the configuration. |