MishActivation<T>

Models & Types · AiDotNet.ActivationFunctions

Implements the Mish activation function for neural networks.

Inheritance: ActivationFunctionBase<T> → MishActivation<T>

Implements: IFusedActivation

For Beginners

The Mish activation function is a smooth, non-monotonic function that helps neural networks learn complex patterns. It was introduced in 2019 and has shown good performance in many applications.

Mathematically, Mish is defined as: f(x) = x * tanh(softplus(x)) where softplus(x) = ln(1 + e^x)

Mish combines properties of several popular activation functions:

  • It’s smooth (no sharp corners like ReLU)
  • It allows both positive and negative values (unlike ReLU which zeros out negatives)
  • It’s unbounded on the positive side (can output large positive values)
  • It’s bounded on the negative side (won’t output extremely negative values)

These properties help neural networks learn more effectively in many situations.

Example

using AiDotNet.ActivationFunctions;

var activation = new MishActivation<double>();
double y = activation.Activate(0.5);
Console.WriteLine($"MishActivation: f(0.5) = {y:F4}");

Constructors

Constructor Summary
MishActivation()

Properties

Property Summary
bool SupportsGpuTraining { get; } Gets whether Mish supports GPU-resident training.

Methods

Method Summary
T Activate(T input) Applies Mish to a tensor via the engine so the gradient tape records the op.
T Derivative(T input) Calculates the derivative (gradient) of the Mish function for a single input value.
Tensor<T> Activate(Tensor<T> input) Applies Mish to a tensor via the engine so the gradient tape records the op.
void ForwardGpu(IDirectGpuBackend backend, IGpuBuffer input, IGpuBuffer output, int size) Applies the Mish activation function on GPU.