ReLU6Activation<T>

Models & Types · AiDotNet.ActivationFunctions

Implements the ReLU6 (Rectified Linear Unit capped at 6) activation function.

Inheritance: ActivationFunctionBase<T> → ReLU6Activation<T>

Implements: IFusedActivation

For Beginners

ReLU6 works like regular ReLU but adds an upper limit of 6.

While ReLU allows any positive value to pass through unchanged, ReLU6 caps the output at 6:

  • Negative inputs become 0 (same as ReLU)
  • Values between 0 and 6 pass through unchanged
  • Values above 6 are capped at 6

This is particularly useful in mobile neural networks (like MobileNet) because:

  • It prevents activations from becoming too large, improving numerical stability
  • It works well with low-precision arithmetic (like 8-bit integers) on mobile devices
  • It helps the network learn more robust features by limiting extreme activations

How It Works

ReLU6 is a variant of the ReLU activation function that clips the output at 6. It is defined as: f(x) = min(max(0, x), 6)

Example

using AiDotNet.ActivationFunctions;

var activation = new ReLU6Activation<double>();
double y = activation.Activate(0.5);
Console.WriteLine($"ReLU6Activation: f(0.5) = {y:F4}");

Constructors

Constructor Summary
ReLU6Activation() Initializes a new instance of the ReLU6Activation class.

Methods

Method Summary
Matrix<T> Derivative(Vector<T> input) Calculates the derivative of the ReLU6 function for each element in a tensor.
T Activate(T input) Applies the ReLU6 activation function to each element in a tensor through the engine’s ReLU6 op, which the gradient tape records with its own backward (gradient 1 where 0 < x < 6, as PyTorch’s ReLU6).
T Derivative(T input) Calculates the derivative of the ReLU6 function for each element in a tensor.
Tensor<T> Activate(Tensor<T> input) Applies the ReLU6 activation function to each element in a tensor through the engine’s ReLU6 op, which the gradient tape records with its own backward (gradient 1 where 0 < x < 6, as PyTorch’s ReLU6).
Tensor<T> Derivative(Tensor<T> input) Calculates the derivative of the ReLU6 function for each element in a tensor.
Vector<T> Activate(Vector<T> input) Applies the ReLU6 activation function to each element in a tensor through the engine’s ReLU6 op, which the gradient tape records with its own backward (gradient 1 where 0 < x < 6, as PyTorch’s ReLU6).