LossScaler<T>

Models & Types · AiDotNet.MixedPrecision

Implements dynamic loss scaling for mixed-precision training to prevent gradient underflow.

For Beginners

Loss scaling is a technique used in mixed-precision training to prevent very small gradient values from becoming zero (underflow) when using 16-bit precision.

The problem:

  • FP16 (Half) can only represent numbers in the range [6e-8, 65504]
  • During training, gradients are often very small (e.g., 1e-10)
  • Small gradients underflow to zero in FP16, stopping learning

The solution:

  • Scale the loss by a large factor (e.g., 2^16 = 65536) before backpropagation
  • This makes gradients larger, preventing underflow
  • Unscale gradients back to their original values before parameter updates

Dynamic scaling:

  • Automatically adjusts the scale factor during training
  • Increases scale when gradients are stable (no overflow)
  • Decreases scale when gradients overflow (become infinity/NaN)

How It Works

Technical Details: The algorithm follows NVIDIA’s approach:

  1. Start with a large initial scale (default: 2^16 = 65536)
  2. If no overflow for N steps, increase scale by growth factor (default: 2.0)
  3. If overflow detected, decrease scale by backoff factor (default: 0.5) and skip update
  4. Monitor consecutive successful updates for scale adjustment

Example

var scaler = new LossScaler<float>(
    initialScale: 65536.0,
    dynamicScaling: true
);
            
// Scale the loss before the backward pass, so gradients too small to survive fp16 stay representable.
float loss = 0.25f;
float scaledLoss = scaler.ScaleLoss(loss);
            
// After the backward pass, check the gradients before trusting them.
var gradients = Tensor<float>.CreateRandom(2, 4);
if (scaler.DetectOverflow(gradients))
{
    // Overflow: skip this update. A dynamic scaler backs its scale off on its own.
}
else
{
    scaler.UnscaleGradients(gradients);
    // Safe to update parameters
}
            
Console.WriteLine($"scale {scaler.Scale}, overflow rate {scaler.OverflowRate:P1}");

Constructors

Constructor Summary
LossScaler(double initialScale, bool dynamicScaling, int growthInterval, double growthFactor, double backoffFactor, double minScale, double maxScale) Initializes a new instance of the LossScaler class.

Properties

Property Summary
bool DynamicScaling { get; set; } Whether to use dynamic loss scaling.
double BackoffFactor { get; set; } Factor by which to multiply the scale when decreasing (default: 0.5).
double GrowthFactor { get; set; } Factor by which to multiply the scale when increasing (default: 2.0).
double MaxScale { get; set; } Maximum allowed scale value to prevent excessive growth.
double MinScale { get; set; } Minimum allowed scale value to prevent excessive reduction.
double OverflowRate { get; } Gets the overflow rate (skipped / total).
double Scale { get; set; } Current loss scale factor.
int GrowthInterval { get; set; } Number of consecutive iterations without overflow before increasing scale.
int SkippedUpdates { get; } Gets the number of updates skipped due to overflow.
int TotalUpdates { get; } Gets the total number of updates attempted.

Methods

Method Summary
T ScaleLoss(T loss) Scales the loss value to prevent gradient underflow.
T UnscaleGradient(T gradient) Unscales a single gradient value.
bool DetectOverflow(Tensor<T> gradients) Checks if any gradient in a vector has overflowed.
bool DetectOverflow(Vector<T> gradients) Checks if any gradient in a vector has overflowed.
bool HasOverflow(T value) Checks if a single value has overflowed (is NaN or infinity).
bool UnscaleGradientsAndCheck(Tensor<T> gradients) Unscales gradients and checks for overflow (vector version).
bool UnscaleGradientsAndCheck(Vector<T> gradients) Unscales gradients and checks for overflow (vector version).
string ToString() Gets a summary of the loss scaler’s current state.
void RecordOverflow() Records that the current optimizer step OVERFLOWED — equivalent to UnscaleGradientsAndCheck finding NaN/Inf in the scaled gradients, but without touching any gradient buffer.
void RecordSuccess() Records that the current optimizer step succeeded — equivalent to UnscaleGradientsAndCheck finding no NaN/Inf in the scaled gradients, but without touching any gradient buffer.
void Reset(double? newInitialScale) Resets the statistics and scale to initial values.
void UnscaleGradients(Tensor<T> gradients) Unscales all gradients in a vector.
void UnscaleGradients(Vector<T> gradients) Unscales all gradients in a vector.