LayerNormalizationLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Represents a Layer Normalization layer that normalizes inputs across the feature dimension.

Inheritance: LayerBase<T> → LayerNormalizationLayer<T>

Implements: IShapeContract

For Beginners

This layer helps stabilize and speed up training by standardizing the data.

Think of Layer Normalization like standardizing test scores:

  • It makes each sample’s features have a mean of 0 and standard deviation of 1
  • It does this independently for each sample (unlike Batch Normalization)
  • It applies this normalization along the feature dimension
  • After normalizing, it scales and shifts the values using learnable parameters

For example, in a sentiment analysis task, some input sentences might use very positive words while others use more neutral language. Layer Normalization helps the network focus on the relative importance of features within each sample rather than their absolute values.

This is particularly useful for:

  • Recurrent neural networks
  • Cases where batch sizes are small
  • Making training more stable and faster

How It Works

Layer Normalization is a technique used to normalize the inputs to a layer, which can help improve training stability and speed. Unlike Batch Normalization which normalizes across the batch dimension, Layer Normalization normalizes across the feature dimension independently for each sample. This makes it particularly useful for recurrent networks and when batch sizes are small. The layer learns scale (gamma) and shift (beta) parameters to allow the network to recover the original representation if needed.

Constructors

Constructor Summary
LayerNormalizationLayer(double epsilon) Initializes a new instance of the LayerNormalizationLayer class with the specified feature size and epsilon value.
LayerNormalizationLayer(int featureSize, double epsilon) AiDotNet#1370 eager-init constructor.

Properties

Property Summary
bool MakesUpstreamBiasRedundant { get; } Gets a value indicating whether this layer supports training.
bool SupportsTraining { get; }

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
T GetEpsilon() Gets the epsilon value used for numerical stability.
Tensor<T> GetBetaTensor() Gets the beta tensor for JIT compilation and internal use.
Tensor<T> GetGammaTensor() Gets the gamma tensor for JIT compilation and internal use.
Vector<T> GetBeta() Gets the beta (shift) parameters of the layer normalization layer.
Vector<T> GetGamma() Gets the gamma (scale) parameters of the layer normalization layer.
int[] GetNormalizedShape() Gets the normalized shape (feature size) of the layer.
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate) Updates the parameters of the layer using the calculated gradients.
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).