SiLUActivation<T>
Models & Types ยท AiDotNet.ActivationFunctions
Implements the SiLU (Sigmoid Linear Unit) activation function, also known as Swish.
Inheritance: ActivationFunctionBase<T> โ SiLUActivation<T>
Implements: IFusedActivation
For Beginners
SiLU (or Swish) is a relatively new activation function that has become popular in modern neural networks. Unlike simpler functions like ReLU that either pass a value through or block it, SiLU smoothly scales inputs based on their value. It keeps most positive values, reduces small positive values, and allows some negative values to pass through (but reduced in magnitude). This smooth behavior helps neural networks learn more complex patterns. SiLU is used in many state-of-the-art models, especially in deep learning applications like computer vision and natural language processing.
How It Works
The SiLU function is defined as f(x) = x * sigmoid(x), where sigmoid(x) = 1/(1+e^(-x)). It was introduced in 2017 and has shown strong performance in deep neural networks.
Example
using AiDotNet.ActivationFunctions;
var activation = new SiLUActivation<double>();
double y = activation.Activate(0.5);
Console.WriteLine($"SiLUActivation: f(0.5) = {y:F4}");
Constructors
| Constructor | Summary |
|---|---|
SiLUActivation() |
Properties
| Property | Summary |
|---|---|
bool SupportsGpuTraining { get; } |
Gets whether SiLU supports GPU-resident training. |
Methods
| Method | Summary |
|---|---|
T Activate(T input) |
Applies SiLU/Swish to a tensor via the engine so the gradient tape records it. |
T Derivative(T input) |
Calculates the derivative of the SiLU function for a single input value. |
Tensor<T> Activate(Tensor<T> input) |
Applies SiLU/Swish to a tensor via the engine so the gradient tape records it. |
void ForwardGpu(IDirectGpuBackend backend, IGpuBuffer input, IGpuBuffer output, int size) |
Applies the SiLU activation function on GPU. |