HardSwishActivation<T>

Models & Types ยท AiDotNet.ActivationFunctions

Implements the Hard Swish activation function used in MobileNetV3.

Inheritance: ActivationFunctionBase<T> โ†’ HardSwishActivation<T>

Implements: IFusedActivation

For Beginners

Hard Swish combines the benefits of ReLU and Sigmoid-like activations.

The Swish function (x * sigmoid(x)) performs well in neural networks but requires computing the expensive sigmoid function. Hard Swish approximates this using simpler operations:

  • For inputs < -3: output is 0 (similar to ReLU for negative values)
  • For inputs > 3: output equals the input (similar to identity for large positive values)
  • For inputs between -3 and 3: smooth transition using a simple piecewise linear function

Hard Swish is particularly important in MobileNetV3 because:

  • It is faster to compute than regular Swish on mobile devices
  • It provides better accuracy than ReLU for deeper networks
  • It is compatible with quantized inference (8-bit integer arithmetic)

How It Works

Hard Swish is a computationally efficient approximation of the Swish activation function. It is defined as: f(x) = x * min(max(0, x + 3), 6) / 6

Example

using AiDotNet.ActivationFunctions;

var activation = new HardSwishActivation<double>();
double y = activation.Activate(0.5);
Console.WriteLine($"HardSwishActivation: f(0.5) = {y:F4}");

Constructors

Constructor Summary
HardSwishActivation() Initializes a new instance of the HardSwishActivation class.

Properties

Property Summary
bool SupportsGpuTraining { get; } Gets whether HardSwish supports GPU-resident training.

Methods

Method Summary
Matrix<T> Derivative(Vector<T> input) Calculates the derivative of the Hard Swish function for each element in a tensor.
T Activate(T input) Applies the Hard Swish activation function to each element in a tensor.
T Derivative(T input) Calculates the derivative of the Hard Swish function for each element in a tensor.
Tensor<T> Activate(Tensor<T> input) Applies the Hard Swish activation function to each element in a tensor.
Tensor<T> Derivative(Tensor<T> input) Calculates the derivative of the Hard Swish function for each element in a tensor.
Vector<T> Activate(Vector<T> input) Applies the Hard Swish activation function to each element in a tensor.
void ForwardGpu(IDirectGpuBackend backend, IGpuBuffer input, IGpuBuffer output, int size) Applies the HardSwish activation function on GPU.