GroupNormalizationLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
Represents a Group Normalization layer that normalizes inputs across groups of channels.
Inheritance: LayerBase<T> → GroupNormalizationLayer<T>
Implements: IShapeContract
For Beginners
This layer helps stabilize training for convolutional networks.
Think of Group Normalization like organizing students into study groups:
- Each group (of channels) studies together and normalizes their behavior
- It works the same regardless of class size (batch size)
- This is especially useful for generative models like VAEs where batch sizes may be small
Key advantages:
- Works well with small batch sizes (even batch size of 1)
- More stable than Batch Normalization for generative models
- Used extensively in modern architectures like Stable Diffusion VAE
Typical usage:
- numGroups=32 for 256+ channels
- numGroups=16 for 128 channels
- numGroups=8 for 64 channels
How It Works
Group Normalization divides channels into groups and normalizes the features within each group. This makes it invariant to batch size, making it suitable for small batch sizes or applications where batch statistics are not reliable (like VAEs and generative models).
Constructors
| Constructor | Summary |
|---|---|
GroupNormalizationLayer(int numGroups, int numChannels, double epsilon) |
Properties
| Property | Summary |
|---|---|
bool MakesUpstreamBiasRedundant { get; } |
|
bool SupportsGpuTraining { get; } |
Gets a value indicating whether this layer supports GPU-resident training. |
bool SupportsTraining { get; } |
|
int NumChannels { get; } |
|
int NumGroups { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
T GetEpsilon() |
|
Tensor<T> GetBetaTensor() |
|
Tensor<T> GetGammaTensor() |
|
Vector<T> GetBeta() |
|
Vector<T> GetGamma() |
|
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void UpdateParametersGpu(IGpuOptimizerConfig config) |
GPU-resident parameter update using the provided optimizer configuration. |
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |