ThresholdedReLUActivation<T>

Models & Types · AiDotNet.ActivationFunctions

Implements the Thresholded ReLU activation function, a variant of the standard ReLU function with an adjustable threshold.

Inheritance: ActivationFunctionBase<T> → ThresholdedReLUActivation<T>

For Beginners

The Thresholded ReLU (Rectified Linear Unit) is a variation of the standard ReLU activation function.

While a standard ReLU outputs the input value when it’s positive and zero when it’s negative (f(x) = max(0, x)), the Thresholded ReLU adds an additional parameter called “theta” (?) that acts as a threshold.

The Thresholded ReLU only activates (returns the input value) when the input exceeds this threshold. Otherwise, it returns zero. The formula is:

f(x) = x if x > ?, otherwise f(x) = 0

This allows the neural network to ignore small positive activations that might be noise, potentially creating more robust models. By adjusting the threshold value, you can control how sensitive the activation function is to input signals.

Example

using AiDotNet.ActivationFunctions;

var activation = new ThresholdedReLUActivation<double>();
double y = activation.Activate(0.5);
Console.WriteLine($"ThresholdedReLUActivation: f(0.5) = {y:F4}");

Constructors

Constructor Summary
ThresholdedReLUActivation(double theta) Initializes a new instance of the ThresholdedReLUActivation class with the specified threshold value.

Methods

Method Summary
T Activate(T input) Applies ThresholdedReLU to a tensor via engine primitives so the gradient tape records every step.
T Derivative(T input) Calculates the derivative of the Thresholded ReLU function for a given input.
Tensor<T> Activate(Tensor<T> input) Applies ThresholdedReLU to a tensor via engine primitives so the gradient tape records every step.
void UpdateTheta(T newTheta) Updates the threshold value used by the activation function.