LossScaler<T>
Models & Types · AiDotNet.MixedPrecision
Implements dynamic loss scaling for mixed-precision training to prevent gradient underflow.
For Beginners
Loss scaling is a technique used in mixed-precision training to prevent very small gradient values from becoming zero (underflow) when using 16-bit precision.
The problem:
- FP16 (Half) can only represent numbers in the range [6e-8, 65504]
- During training, gradients are often very small (e.g., 1e-10)
- Small gradients underflow to zero in FP16, stopping learning
The solution:
- Scale the loss by a large factor (e.g., 2^16 = 65536) before backpropagation
- This makes gradients larger, preventing underflow
- Unscale gradients back to their original values before parameter updates
Dynamic scaling:
- Automatically adjusts the scale factor during training
- Increases scale when gradients are stable (no overflow)
- Decreases scale when gradients overflow (become infinity/NaN)
How It Works
Technical Details: The algorithm follows NVIDIA’s approach:
- Start with a large initial scale (default: 2^16 = 65536)
- If no overflow for N steps, increase scale by growth factor (default: 2.0)
- If overflow detected, decrease scale by backoff factor (default: 0.5) and skip update
- Monitor consecutive successful updates for scale adjustment
Example
var scaler = new LossScaler<float>(
initialScale: 65536.0,
dynamicScaling: true
);
// Scale the loss before the backward pass, so gradients too small to survive fp16 stay representable.
float loss = 0.25f;
float scaledLoss = scaler.ScaleLoss(loss);
// After the backward pass, check the gradients before trusting them.
var gradients = Tensor<float>.CreateRandom(2, 4);
if (scaler.DetectOverflow(gradients))
{
// Overflow: skip this update. A dynamic scaler backs its scale off on its own.
}
else
{
scaler.UnscaleGradients(gradients);
// Safe to update parameters
}
Console.WriteLine($"scale {scaler.Scale}, overflow rate {scaler.OverflowRate:P1}");
Constructors
| Constructor | Summary |
|---|---|
LossScaler(double initialScale, bool dynamicScaling, int growthInterval, double growthFactor, double backoffFactor, double minScale, double maxScale) |
Initializes a new instance of the LossScaler class. |
Properties
| Property | Summary |
|---|---|
bool DynamicScaling { get; set; } |
Whether to use dynamic loss scaling. |
double BackoffFactor { get; set; } |
Factor by which to multiply the scale when decreasing (default: 0.5). |
double GrowthFactor { get; set; } |
Factor by which to multiply the scale when increasing (default: 2.0). |
double MaxScale { get; set; } |
Maximum allowed scale value to prevent excessive growth. |
double MinScale { get; set; } |
Minimum allowed scale value to prevent excessive reduction. |
double OverflowRate { get; } |
Gets the overflow rate (skipped / total). |
double Scale { get; set; } |
Current loss scale factor. |
int GrowthInterval { get; set; } |
Number of consecutive iterations without overflow before increasing scale. |
int SkippedUpdates { get; } |
Gets the number of updates skipped due to overflow. |
int TotalUpdates { get; } |
Gets the total number of updates attempted. |
Methods
| Method | Summary |
|---|---|
T ScaleLoss(T loss) |
Scales the loss value to prevent gradient underflow. |
T UnscaleGradient(T gradient) |
Unscales a single gradient value. |
bool DetectOverflow(Tensor<T> gradients) |
Checks if any gradient in a vector has overflowed. |
bool DetectOverflow(Vector<T> gradients) |
Checks if any gradient in a vector has overflowed. |
bool HasOverflow(T value) |
Checks if a single value has overflowed (is NaN or infinity). |
bool UnscaleGradientsAndCheck(Tensor<T> gradients) |
Unscales gradients and checks for overflow (vector version). |
bool UnscaleGradientsAndCheck(Vector<T> gradients) |
Unscales gradients and checks for overflow (vector version). |
string ToString() |
Gets a summary of the loss scaler’s current state. |
void RecordOverflow() |
Records that the current optimizer step OVERFLOWED — equivalent to UnscaleGradientsAndCheck finding NaN/Inf in the scaled gradients, but without touching any gradient buffer. |
void RecordSuccess() |
Records that the current optimizer step succeeded — equivalent to UnscaleGradientsAndCheck finding no NaN/Inf in the scaled gradients, but without touching any gradient buffer. |
void Reset(double? newInitialScale) |
Resets the statistics and scale to initial values. |
void UnscaleGradients(Tensor<T> gradients) |
Unscales all gradients in a vector. |
void UnscaleGradients(Vector<T> gradients) |
Unscales all gradients in a vector. |