MixedPrecisionConfig

Options & Configuration · AiDotNet.MixedPrecision

Configuration settings for mixed-precision training.

For Beginners

This class contains all the settings you can adjust for mixed-precision training. The default values work well for most models, but you can customize them based on your specific needs.

Key concepts:

  • Loss Scaling: Prevents small gradients from becoming zero in FP16
  • Dynamic Scaling: Automatically adjusts the loss scale during training
  • Master Weights: FP32 copy of parameters for precise updates
  • Working Weights: FP16 copy used for forward/backward passes
  • FP8 (New): 8-bit formats for 2x throughput on H100+ GPUs

Constructors

Constructor Summary
MixedPrecisionConfig() Creates a configuration with default recommended settings.

Properties

Property Summary
List<string> FP8ExcludedLayers { get; set; } Layers to keep in higher precision (FP16/BF16) even when using FP8.
MixedPrecisionType FP8BackwardFormat { get; set; } Format to use for backward pass in FP8 mode (default: E5M2).
MixedPrecisionType FP8ForwardFormat { get; set; } Format to use for forward pass in FP8 mode (default: E4M3).
MixedPrecisionType PrecisionType { get; set; } The precision type to use for mixed-precision training (default: FP16).
bool EnableDynamicScaling { get; set; } Enable dynamic loss scaling (default: true).
bool FP8PerTensorScaling { get; set; } Whether to use per-tensor scaling for FP8 (default: true).
bool Fp32BatchNorm { get; set; } Whether to keep batch normalization layers in FP32 (default: true).
bool Fp32GradientAccumulation { get; set; } Whether to accumulate gradients in FP32 (default: true).
bool Fp32Loss { get; set; } Whether to keep loss computation in FP32 (default: true).
double InitialLossScale { get; set; } Initial loss scale factor (default: 65536 = 2^16).
double MaxLossScale { get; set; } Maximum allowed loss scale (default: 16777216 = 2^24).
double MinLossScale { get; set; } Minimum allowed loss scale (default: 1.0).
double ScaleBackoffFactor { get; set; } Factor by which to multiply scale when decreasing (default: 0.5).
double ScaleGrowthFactor { get; set; } Factor by which to multiply scale when increasing (default: 2.0).
int ScaleGrowthInterval { get; set; } Number of consecutive successful updates before increasing scale (default: 2000).

Methods

Method Summary
MixedPrecisionConfig Aggressive() Creates an aggressive configuration optimized for maximum throughput.
MixedPrecisionConfig Conservative() Creates a conservative configuration optimized for training stability.
MixedPrecisionConfig ForBF16() Creates a configuration for BF16 precision (Ampere+ GPUs).
MixedPrecisionConfig ForFP8() Creates a configuration for FP8 hybrid mode (H100+ GPUs).
MixedPrecisionConfig ForFP8Transformers() Creates a configuration for FP8 training on transformer models.
string ToString() Gets a summary of the configuration.