LeakyReLUActivation<T>

Models & Types · AiDotNet.ActivationFunctions

Implements the Leaky Rectified Linear Unit (Leaky ReLU) activation function for neural networks.

Inheritance: ActivationFunctionBase<T> → LeakyReLUActivation<T>

Implements: IFusedActivation

For Beginners

The Leaky ReLU activation function is a variation of the standard ReLU function.

How it works:

  • For positive inputs (x > 0): It returns the input unchanged (like a straight line)
  • For negative inputs (x = 0): It returns a small fraction of the input (a * x)

The main advantage of Leaky ReLU over standard ReLU is that it never completely “turns off” neurons for negative inputs. Instead, it allows a small gradient to flow through, which helps prevent the “dying ReLU” problem where neurons can stop learning during training.

Think of it like a water pipe that:

  • Allows full flow when the input is positive
  • Allows a small “leak” when the input is negative (controlled by the alpha parameter)

Example

using AiDotNet.ActivationFunctions;

var activation = new LeakyReLUActivation<double>();
double y = activation.Activate(0.5);
Console.WriteLine($"LeakyReLUActivation: f(0.5) = {y:F4}");

Constructors

Constructor Summary
LeakyReLUActivation() Initializes a new instance of the Leaky ReLU activation function with the default slope (alpha = 0.01).
LeakyReLUActivation(double alpha) Initializes a new instance of the Leaky ReLU activation function with the specified alpha parameter.

Properties

Property Summary
T Alpha { get; } Gets the slope coefficient for negative input values.
bool SupportsGpuTraining { get; } Gets whether LeakyReLU supports GPU-resident training.

Methods

Method Summary
Matrix<T> Derivative(Vector<T> input) Calculates the derivative of the Leaky ReLU function for each element in a tensor.
T Activate(T input) Applies the Leaky ReLU activation function to each element in a tensor.
T Derivative(T input) Calculates the derivative of the Leaky ReLU function for each element in a tensor.
Tensor<T> Activate(Tensor<T> input) Applies the Leaky ReLU activation function to each element in a tensor.
Tensor<T> Derivative(Tensor<T> input) Calculates the derivative of the Leaky ReLU function for each element in a tensor.
Vector<T> Activate(Vector<T> input) Applies the Leaky ReLU activation function to each element in a tensor.
void ForwardGpu(IDirectGpuBackend backend, IGpuBuffer input, IGpuBuffer output, int size) Applies the Leaky ReLU activation function on GPU.