SiLUActivation<T>

Models & Types ยท AiDotNet.ActivationFunctions

Implements the SiLU (Sigmoid Linear Unit) activation function, also known as Swish.

Inheritance: ActivationFunctionBase<T> โ†’ SiLUActivation<T>

Implements: IFusedActivation

For Beginners

SiLU (or Swish) is a relatively new activation function that has become popular in modern neural networks. Unlike simpler functions like ReLU that either pass a value through or block it, SiLU smoothly scales inputs based on their value. It keeps most positive values, reduces small positive values, and allows some negative values to pass through (but reduced in magnitude). This smooth behavior helps neural networks learn more complex patterns. SiLU is used in many state-of-the-art models, especially in deep learning applications like computer vision and natural language processing.

How It Works

The SiLU function is defined as f(x) = x * sigmoid(x), where sigmoid(x) = 1/(1+e^(-x)). It was introduced in 2017 and has shown strong performance in deep neural networks.

Example

using AiDotNet.ActivationFunctions;

var activation = new SiLUActivation<double>();
double y = activation.Activate(0.5);
Console.WriteLine($"SiLUActivation: f(0.5) = {y:F4}");

Constructors

Constructor Summary
SiLUActivation()

Properties

Property Summary
bool SupportsGpuTraining { get; } Gets whether SiLU supports GPU-resident training.

Methods

Method Summary
T Activate(T input) Applies SiLU/Swish to a tensor via the engine so the gradient tape records it.
T Derivative(T input) Calculates the derivative of the SiLU function for a single input value.
Tensor<T> Activate(Tensor<T> input) Applies SiLU/Swish to a tensor via the engine so the gradient tape records it.
void ForwardGpu(IDirectGpuBackend backend, IGpuBuffer input, IGpuBuffer output, int size) Applies the SiLU activation function on GPU.