LeakyReLUActivation<T>
Models & Types · AiDotNet.ActivationFunctions
Implements the Leaky Rectified Linear Unit (Leaky ReLU) activation function for neural networks.
Inheritance: ActivationFunctionBase<T> → LeakyReLUActivation<T>
Implements: IFusedActivation
For Beginners
The Leaky ReLU activation function is a variation of the standard ReLU function.
How it works:
- For positive inputs (x > 0): It returns the input unchanged (like a straight line)
- For negative inputs (x = 0): It returns a small fraction of the input (a * x)
The main advantage of Leaky ReLU over standard ReLU is that it never completely “turns off” neurons for negative inputs. Instead, it allows a small gradient to flow through, which helps prevent the “dying ReLU” problem where neurons can stop learning during training.
Think of it like a water pipe that:
- Allows full flow when the input is positive
- Allows a small “leak” when the input is negative (controlled by the alpha parameter)
Example
using AiDotNet.ActivationFunctions;
var activation = new LeakyReLUActivation<double>();
double y = activation.Activate(0.5);
Console.WriteLine($"LeakyReLUActivation: f(0.5) = {y:F4}");
Constructors
| Constructor | Summary |
|---|---|
LeakyReLUActivation() |
Initializes a new instance of the Leaky ReLU activation function with the default slope (alpha = 0.01). |
LeakyReLUActivation(double alpha) |
Initializes a new instance of the Leaky ReLU activation function with the specified alpha parameter. |
Properties
| Property | Summary |
|---|---|
T Alpha { get; } |
Gets the slope coefficient for negative input values. |
bool SupportsGpuTraining { get; } |
Gets whether LeakyReLU supports GPU-resident training. |
Methods
| Method | Summary |
|---|---|
Matrix<T> Derivative(Vector<T> input) |
Calculates the derivative of the Leaky ReLU function for each element in a tensor. |
T Activate(T input) |
Applies the Leaky ReLU activation function to each element in a tensor. |
T Derivative(T input) |
Calculates the derivative of the Leaky ReLU function for each element in a tensor. |
Tensor<T> Activate(Tensor<T> input) |
Applies the Leaky ReLU activation function to each element in a tensor. |
Tensor<T> Derivative(Tensor<T> input) |
Calculates the derivative of the Leaky ReLU function for each element in a tensor. |
Vector<T> Activate(Vector<T> input) |
Applies the Leaky ReLU activation function to each element in a tensor. |
void ForwardGpu(IDirectGpuBackend backend, IGpuBuffer input, IGpuBuffer output, int size) |
Applies the Leaky ReLU activation function on GPU. |