ISRUActivation<T>

Models & Types · AiDotNet.ActivationFunctions

Implements the Inverse Square Root Unit (ISRU) activation function for neural networks.

Inheritance: ActivationFunctionBase<T> → ISRUActivation<T>

For Beginners

The ISRU (Inverse Square Root Unit) activation function is designed to:

  • Keep the output values bounded (they never grow too large)
  • Preserve the sign of the input (positive inputs give positive outputs, negative inputs give negative outputs)
  • Allow gradients to flow more easily during training compared to some other functions

The function looks like an “S” shape that’s been stretched horizontally, similar to tanh but with different mathematical properties. It approaches +1 for large positive inputs and -1 for large negative inputs, but never quite reaches these values.

The a (alpha) parameter controls how quickly the function “saturates” (flattens out):

  • Smaller a values make the function change more gradually
  • Larger a values make the function change more abruptly

ISRU is useful in neural networks where you want bounded outputs but need to avoid the vanishing gradient problem that affects some other activation functions.

Example

using AiDotNet.ActivationFunctions;

var activation = new ISRUActivation<double>();
double y = activation.Activate(0.5);
Console.WriteLine($"ISRUActivation: f(0.5) = {y:F4}");

Constructors

Constructor Summary
ISRUActivation(double alpha) Initializes a new instance of the ISRU activation function with the specified alpha parameter.

Methods

Method Summary
T Activate(T input) Applies ISRU to a tensor via engine primitives so the gradient tape records every step of x / sqrt(1 + αx²).
T Derivative(T input) Calculates the derivative (gradient) of the ISRU function for a single value.
Tensor<T> Activate(Tensor<T> input) Applies ISRU to a tensor via engine primitives so the gradient tape records every step of x / sqrt(1 + αx²).