RReLUActivation<T>

Models & Types · AiDotNet.ActivationFunctions

Implements the Randomized Rectified Linear Unit (RReLU) activation function.

Inheritance: ActivationFunctionBase<T> → RReLUActivation<T>

For Beginners

RReLU (Randomized Rectified Linear Unit) is like a “smart filter” for neural networks. When data is positive, it lets it pass through unchanged. When data is negative, instead of setting it to zero (like regular ReLU), it reduces the value by multiplying it by a small random number. This randomness helps prevent “dead neurons” (neurons that stop learning) and can improve the network’s ability to learn. During training, this random factor changes; during testing/inference, a fixed average value is used.

How It Works

RReLU is a variation of the ReLU activation function that introduces randomness during training. For positive inputs, it behaves like standard ReLU (returning the input unchanged). For negative inputs, it multiplies the input by a random factor (alpha) between the specified lower and upper bounds.

Example

using AiDotNet.ActivationFunctions;

var activation = new RReLUActivation<double>();
double y = activation.Activate(0.5);
Console.WriteLine($"RReLUActivation: f(0.5) = {y:F4}");

Constructors

Constructor Summary
RReLUActivation(double lowerBound, double upperBound) Initializes a new instance of the RReLU activation function with specified bounds for the random alpha parameter.

Methods

Method Summary
T Activate(T input) Applies RReLU to a tensor via engine primitives so the gradient tape records every step.
T Derivative(T input) Calculates the derivative of the RReLU function for a single input value.
Tensor<T> Activate(Tensor<T> input) Applies RReLU to a tensor via engine primitives so the gradient tape records every step.
void SetTrainingMode(bool isTraining) Sets the activation function to either training or inference mode.