SoftSignActivation<T>
Models & Types · AiDotNet.ActivationFunctions
Implements the SoftSign activation function, which is a smooth alternative to the tanh function.
Inheritance: ActivationFunctionBase<T> → SoftSignActivation<T>
Implements: IFusedActivation
For Beginners
SoftSign is an activation function that squeezes input values into a range between -1 and 1. It’s similar to the tanh function but approaches its limits more slowly.
Key properties of SoftSign:
- For input of 0, the output is 0
- For large positive inputs, the output approaches 1 (but never reaches it)
- For large negative inputs, the output approaches -1 (but never reaches it)
- The function is smooth everywhere, making it easier to train
- Unlike tanh, SoftSign has “polynomial” tails, meaning it approaches its limits more gradually
This gradual approach to limits can help prevent neurons from becoming “saturated” (stuck at extreme values) during training, which can be an advantage in some neural network architectures.
How It Works
The SoftSign function is defined as: f(x) = x / (1 + |x|), where |x| is the absolute value of x. It maps any input value to an output between -1 and 1, similar to tanh but with different properties.
Example
using AiDotNet.ActivationFunctions;
var activation = new SoftSignActivation<double>();
double y = activation.Activate(0.5);
Console.WriteLine($"SoftSignActivation: f(0.5) = {y:F4}");
Constructors
| Constructor | Summary |
|---|---|
SoftSignActivation() |
Methods
| Method | Summary |
|---|---|
T Activate(T input) |
Applies SoftSign to a tensor via engine primitives so the gradient tape records every step of x / (1 + |x|). |
T Derivative(T input) |
Calculates the derivative of the SoftSign function for a given input. |
Tensor<T> Activate(Tensor<T> input) |
Applies SoftSign to a tensor via engine primitives so the gradient tape records every step of x / (1 + |x|). |