ReLU6Activation<T>
Models & Types · AiDotNet.ActivationFunctions
Implements the ReLU6 (Rectified Linear Unit capped at 6) activation function.
Inheritance: ActivationFunctionBase<T> → ReLU6Activation<T>
Implements: IFusedActivation
For Beginners
ReLU6 works like regular ReLU but adds an upper limit of 6.
While ReLU allows any positive value to pass through unchanged, ReLU6 caps the output at 6:
- Negative inputs become 0 (same as ReLU)
- Values between 0 and 6 pass through unchanged
- Values above 6 are capped at 6
This is particularly useful in mobile neural networks (like MobileNet) because:
- It prevents activations from becoming too large, improving numerical stability
- It works well with low-precision arithmetic (like 8-bit integers) on mobile devices
- It helps the network learn more robust features by limiting extreme activations
How It Works
ReLU6 is a variant of the ReLU activation function that clips the output at 6. It is defined as: f(x) = min(max(0, x), 6)
Example
using AiDotNet.ActivationFunctions;
var activation = new ReLU6Activation<double>();
double y = activation.Activate(0.5);
Console.WriteLine($"ReLU6Activation: f(0.5) = {y:F4}");
Constructors
| Constructor | Summary |
|---|---|
ReLU6Activation() |
Initializes a new instance of the ReLU6Activation class. |
Methods
| Method | Summary |
|---|---|
Matrix<T> Derivative(Vector<T> input) |
Calculates the derivative of the ReLU6 function for each element in a tensor. |
T Activate(T input) |
Applies the ReLU6 activation function to each element in a tensor through the engine’s ReLU6 op, which the gradient tape records with its own backward (gradient 1 where 0 < x < 6, as PyTorch’s ReLU6). |
T Derivative(T input) |
Calculates the derivative of the ReLU6 function for each element in a tensor. |
Tensor<T> Activate(Tensor<T> input) |
Applies the ReLU6 activation function to each element in a tensor through the engine’s ReLU6 op, which the gradient tape records with its own backward (gradient 1 where 0 < x < 6, as PyTorch’s ReLU6). |
Tensor<T> Derivative(Tensor<T> input) |
Calculates the derivative of the ReLU6 function for each element in a tensor. |
Vector<T> Activate(Vector<T> input) |
Applies the ReLU6 activation function to each element in a tensor through the engine’s ReLU6 op, which the gradient tape records with its own backward (gradient 1 where 0 < x < 6, as PyTorch’s ReLU6). |