GroupNormalizationLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Represents a Group Normalization layer that normalizes inputs across groups of channels.

Inheritance: LayerBase<T> → GroupNormalizationLayer<T>

Implements: IShapeContract

For Beginners

This layer helps stabilize training for convolutional networks.

Think of Group Normalization like organizing students into study groups:

  • Each group (of channels) studies together and normalizes their behavior
  • It works the same regardless of class size (batch size)
  • This is especially useful for generative models like VAEs where batch sizes may be small

Key advantages:

  • Works well with small batch sizes (even batch size of 1)
  • More stable than Batch Normalization for generative models
  • Used extensively in modern architectures like Stable Diffusion VAE

Typical usage:

  • numGroups=32 for 256+ channels
  • numGroups=16 for 128 channels
  • numGroups=8 for 64 channels

How It Works

Group Normalization divides channels into groups and normalizes the features within each group. This makes it invariant to batch size, making it suitable for small batch sizes or applications where batch statistics are not reliable (like VAEs and generative models).

Constructors

Constructor Summary
GroupNormalizationLayer(int numGroups, int numChannels, double epsilon)

Properties

Property Summary
bool MakesUpstreamBiasRedundant { get; }
bool SupportsGpuTraining { get; } Gets a value indicating whether this layer supports GPU-resident training.
bool SupportsTraining { get; }
int NumChannels { get; }
int NumGroups { get; }

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
T GetEpsilon()
Tensor<T> GetBetaTensor()
Tensor<T> GetGammaTensor()
Vector<T> GetBeta()
Vector<T> GetGamma()
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void UpdateParametersGpu(IGpuOptimizerConfig config) GPU-resident parameter update using the provided optimizer configuration.
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).