SwishActivation<T>

Models & Types · AiDotNet.ActivationFunctions

Implements the Swish activation function for neural networks.

Inheritance: ActivationFunctionBase<T> → SwishActivation<T>

Implements: IFusedActivation

For Beginners

Swish is a newer activation function developed by researchers at Google. Its formula is x * sigmoid(x), which means it multiplies the input by the sigmoid of the input.

Key characteristics of Swish include:

  • It’s smooth everywhere, unlike ReLU which has a sharp corner at x=0
  • It allows some negative values through, which can help with learning
  • It behaves somewhat like ReLU for positive values, but has a smoother transition
  • It has been shown to outperform ReLU in some deep neural networks

Swish combines some of the best properties of ReLU and sigmoid functions, making it effective for many deep learning applications.

Example

using AiDotNet.ActivationFunctions;

var activation = new SwishActivation<double>();
double y = activation.Activate(0.5);
Console.WriteLine($"SwishActivation: f(0.5) = {y:F4}");

Constructors

Constructor Summary
SwishActivation()

Properties

Property Summary
bool SupportsGpuTraining { get; } Gets whether Swish supports GPU-resident training.

Methods

Method Summary
Matrix<T> Derivative(Vector<T> input) Calculates the derivative of the Swish activation function for each element of an input vector.
T Activate(T x) Applies the Swish activation function to each element of an input vector.
T Derivative(T x) Calculates the derivative of the Swish activation function for each element of an input vector.
Tensor<T> Activate(Tensor<T> input) Applies the Swish activation function to each element of an input vector.
Vector<T> Activate(Vector<T> input) Applies the Swish activation function to each element of an input vector.
void ForwardGpu(IDirectGpuBackend backend, IGpuBuffer input, IGpuBuffer output, int size) Applies the Swish activation function on GPU.