SwishActivation<T>
Models & Types · AiDotNet.ActivationFunctions
Implements the Swish activation function for neural networks.
Inheritance: ActivationFunctionBase<T> → SwishActivation<T>
Implements: IFusedActivation
For Beginners
Swish is a newer activation function developed by researchers at Google. Its formula is x * sigmoid(x), which means it multiplies the input by the sigmoid of the input.
Key characteristics of Swish include:
- It’s smooth everywhere, unlike ReLU which has a sharp corner at x=0
- It allows some negative values through, which can help with learning
- It behaves somewhat like ReLU for positive values, but has a smoother transition
- It has been shown to outperform ReLU in some deep neural networks
Swish combines some of the best properties of ReLU and sigmoid functions, making it effective for many deep learning applications.
Example
using AiDotNet.ActivationFunctions;
var activation = new SwishActivation<double>();
double y = activation.Activate(0.5);
Console.WriteLine($"SwishActivation: f(0.5) = {y:F4}");
Constructors
| Constructor | Summary |
|---|---|
SwishActivation() |
Properties
| Property | Summary |
|---|---|
bool SupportsGpuTraining { get; } |
Gets whether Swish supports GPU-resident training. |
Methods
| Method | Summary |
|---|---|
Matrix<T> Derivative(Vector<T> input) |
Calculates the derivative of the Swish activation function for each element of an input vector. |
T Activate(T x) |
Applies the Swish activation function to each element of an input vector. |
T Derivative(T x) |
Calculates the derivative of the Swish activation function for each element of an input vector. |
Tensor<T> Activate(Tensor<T> input) |
Applies the Swish activation function to each element of an input vector. |
Vector<T> Activate(Vector<T> input) |
Applies the Swish activation function to each element of an input vector. |
void ForwardGpu(IDirectGpuBackend backend, IGpuBuffer input, IGpuBuffer output, int size) |
Applies the Swish activation function on GPU. |