SoftPlusActivation<T>

Models & Types · AiDotNet.ActivationFunctions

Implements the SoftPlus activation function, which is a smooth approximation of the ReLU function.

Inheritance: ActivationFunctionBase<T> → SoftPlusActivation<T>

Implements: IFusedActivation

For Beginners

SoftPlus is like a “softer” version of the popular ReLU activation function. While ReLU outputs exactly 0 for any negative input and keeps positive values unchanged, SoftPlus creates a smooth curve that’s very close to ReLU but without the sharp corner at x=0.

For negative inputs, SoftPlus outputs small positive values (approaching 0). For large positive inputs, SoftPlus outputs values very close to the input itself.

This smoothness can be helpful in some neural networks because it means the function is differentiable everywhere (it has a well-defined slope at every point), which can make training more stable.

How It Works

The SoftPlus function is defined as: f(x) = ln(1 + e^x), where ln is the natural logarithm. It produces output that is always positive and approaches the ReLU function (max(0,x)) but with a smooth transition at x=0.

Example

using AiDotNet.ActivationFunctions;

var activation = new SoftPlusActivation<double>();
double y = activation.Activate(0.5);
Console.WriteLine($"SoftPlusActivation: f(0.5) = {y:F4}");

Constructors

Constructor Summary
SoftPlusActivation()

Properties

Property Summary
bool SupportsGpuTraining { get; } Gets whether SoftPlus supports GPU-resident training.

Methods

Method Summary
T Activate(T input) Applies the SoftPlus activation function to each element in a tensor.
T Derivative(T input) Calculates the derivative of the SoftPlus function for each element in a tensor.
Tensor<T> Activate(Tensor<T> input) Applies the SoftPlus activation function to each element in a tensor.
Tensor<T> Derivative(Tensor<T> input) Calculates the derivative of the SoftPlus function for each element in a tensor.
void ForwardGpu(IDirectGpuBackend backend, IGpuBuffer input, IGpuBuffer output, int size) Applies the SoftPlus activation function on GPU.