GatedLinearUnitLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Represents a Gated Linear Unit (GLU) layer in a neural network that combines linear transformation with multiplicative gating.

Inheritance: LayerBase<T> → GatedLinearUnitLayer<T>

Implements: IShapeContract

Known derived types: BilinearGLUFeedForwardLayer<T>, GeGLUFeedForwardLayer<T>, ReGLUFeedForwardLayer<T>, SwiGLUFeedForwardLayer<T>

For Beginners

A Gated Linear Unit is like a smart filter that controls how much information flows through.

Imagine water flowing through a pipe with an adjustable valve:

  • The water is the input data
  • One part of the layer (linear part) processes the water
  • Another part (gate) controls how much processed water flows through
  • Together they decide “what information is important to keep”

For example, in language processing:

  • The linear transformation might extract features from words
  • The gate might decide which features are relevant to the current context
  • Their combination helps the network focus on important information

GLUs are particularly good at:

  • Controlling information flow through the network
  • Helping gradients flow during training (preventing vanishing gradients)
  • Allowing the network to selectively use information

This selectivity is valuable in many tasks, especially those involving sequences like text or time-series data.

How It Works

A Gated Linear Unit (GLU) is a neural network layer that combines linear transformations with a gating mechanism. It applies two parallel linear transformations to the input: one produces a linear output, and the other produces a gate that controls how much of the linear output passes through. The final output is the element-wise product of the linear output and the activated gate. GLUs were introduced to help with vanishing gradient problems in deep networks and have been particularly effective in natural language processing and sequence modeling tasks.

Constructors

Constructor Summary
GatedLinearUnitLayer(int outputDimension, IActivationFunction<T> gateActivation) Initializes a new instance of the GatedLinearUnitLayer class with a vector activation function.
GatedLinearUnitLayer(int outputDimension, IVectorActivationFunction<T> gateActivation) Initializes a new instance of the GatedLinearUnitLayer class with a vector activation function.

Properties

Property Summary
bool SupportsTraining { get; } Gets a value indicating whether this layer supports training.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate) Updates the weights and biases for both paths using the calculated gradients and the specified learning rate.
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).