SpectralNormalizationLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Represents a spectral normalization layer that normalizes the weights of a layer by their spectral norm.

Inheritance: LayerBase<T> → SpectralNormalizationLayer<T>

Implements: IShapeContract

For Beginners

Spectral normalization keeps layer weights from getting too large.

Key benefits:

  • Stabilizes GAN training by preventing extreme weight values
  • Ensures the discriminator doesn’t become too powerful too quickly
  • Helps prevent mode collapse in GANs
  • Computationally efficient compared to other normalization methods

How it works:

  • Computes the largest singular value of the weight matrix
  • Divides all weights by this value
  • Keeps weights normalized throughout training

Reference: Miyato et al., “Spectral Normalization for Generative Adversarial Networks” (2018)

How It Works

Spectral normalization is a weight normalization technique that constrains the Lipschitz constant of a neural network layer. It does this by dividing the weight matrix by its largest singular value (spectral norm). This technique is particularly effective for stabilizing GAN training.

Constructors

Constructor Summary
SpectralNormalizationLayer(ILayer<T> innerLayer, int powerIterations) Initializes a new instance of the SpectralNormalizationLayer class.

Properties

Property Summary
bool SupportsGpuTraining { get; } Gets a value indicating whether this layer supports GPU-resident training.
bool SupportsTraining { get; }

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
IReadOnlyList<ValueTuple<string, Tensor<T>>> GetRegisteredBuffers()
void ClearGradients()
void ResetState() Resets the internal state of the layer.
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate) Updates the parameters of the inner layer.
void UpdateParametersGpu(IGpuOptimizerConfig config) GPU-resident parameter update using the provided optimizer configuration.
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).