GradientClippingHelper

Helpers & Utilities · AiDotNet.Helpers

Provides gradient clipping utilities to prevent exploding gradients during training.

For Beginners

During neural network training, gradients tell us how to adjust weights. Sometimes gradients become extremely large (“exploding gradients”), which can destabilize training. Gradient clipping limits the magnitude of gradients to keep training stable.

There are two main approaches:

  • Clip by Value: Limits each gradient element to a range (e.g., -1 to 1)
  • Clip by Norm: Scales the entire gradient vector if its norm exceeds a threshold

The “by norm” approach is generally preferred as it preserves gradient direction.

Methods

Method Summary
List<Vector<T>> ClipByGlobalNorm(List<Vector<T>> gradientsList, double maxNorm) Clips gradients by global norm across multiple gradient vectors.
T ComputeGlobalNorm(List<Vector<T>> gradientsList) Computes the global L2 norm across multiple gradient vectors.
T ComputeNorm(Vector<T> gradients) Computes the L2 norm of a gradient vector.
Tensor<T> ClipByNorm(Tensor<T> gradients, double maxNorm) Clips tensor gradients by their L2 norm.
Vector<T> ClipAdaptive(Vector<T> gradients, Vector<T> parameters, double clipRatio) Applies adaptive gradient clipping based on parameter norm.
Vector<T> ClipByNorm(Vector<T> gradients, double maxNorm) Clips tensor gradients by their L2 norm.
Vector<T> ClipByValue(Vector<T> gradients, double maxValue) Clips gradient values to a specified range [-maxValue, maxValue].
bool AreGradientsExploding(Vector<T> gradients, double threshold) Detects if gradients are exploding (have very large values).
bool AreGradientsVanishing(Vector<T> gradients, double threshold) Detects if gradients are vanishing (have very small values).
bool ClipByNormInPlace(Vector<T> gradients, double maxNorm) Clips gradients by their L2 norm in place.
void ClipByValueInPlace(Vector<T> gradients, double maxValue) Clips gradient values to a specified range [-maxValue, maxValue] in place.

Fields

Field Summary
double DefaultMaxNorm Default maximum gradient norm for clipping.
double DefaultMaxValue Default maximum gradient value for value clipping.