HardSwishActivation<T>
Models & Types ยท AiDotNet.ActivationFunctions
Implements the Hard Swish activation function used in MobileNetV3.
Inheritance: ActivationFunctionBase<T> โ HardSwishActivation<T>
Implements: IFusedActivation
For Beginners
Hard Swish combines the benefits of ReLU and Sigmoid-like activations.
The Swish function (x * sigmoid(x)) performs well in neural networks but requires computing the expensive sigmoid function. Hard Swish approximates this using simpler operations:
- For inputs < -3: output is 0 (similar to ReLU for negative values)
- For inputs > 3: output equals the input (similar to identity for large positive values)
- For inputs between -3 and 3: smooth transition using a simple piecewise linear function
Hard Swish is particularly important in MobileNetV3 because:
- It is faster to compute than regular Swish on mobile devices
- It provides better accuracy than ReLU for deeper networks
- It is compatible with quantized inference (8-bit integer arithmetic)
How It Works
Hard Swish is a computationally efficient approximation of the Swish activation function. It is defined as: f(x) = x * min(max(0, x + 3), 6) / 6
Example
using AiDotNet.ActivationFunctions;
var activation = new HardSwishActivation<double>();
double y = activation.Activate(0.5);
Console.WriteLine($"HardSwishActivation: f(0.5) = {y:F4}");
Constructors
| Constructor | Summary |
|---|---|
HardSwishActivation() |
Initializes a new instance of the HardSwishActivation class. |
Properties
| Property | Summary |
|---|---|
bool SupportsGpuTraining { get; } |
Gets whether HardSwish supports GPU-resident training. |
Methods
| Method | Summary |
|---|---|
Matrix<T> Derivative(Vector<T> input) |
Calculates the derivative of the Hard Swish function for each element in a tensor. |
T Activate(T input) |
Applies the Hard Swish activation function to each element in a tensor. |
T Derivative(T input) |
Calculates the derivative of the Hard Swish function for each element in a tensor. |
Tensor<T> Activate(Tensor<T> input) |
Applies the Hard Swish activation function to each element in a tensor. |
Tensor<T> Derivative(Tensor<T> input) |
Calculates the derivative of the Hard Swish function for each element in a tensor. |
Vector<T> Activate(Vector<T> input) |
Applies the Hard Swish activation function to each element in a tensor. |
void ForwardGpu(IDirectGpuBackend backend, IGpuBuffer input, IGpuBuffer output, int size) |
Applies the HardSwish activation function on GPU. |