Activation Functions

42 public types in this namespace, organized by kind.

Models & Types (40)

Type Summary
BentIdentityActivation<T> Implements the Bent Identity activation function for neural networks.
BinarySpikingActivation<T> Implements the Binary Spiking activation function for neural networks, particularly for spiking neural networks.
CELUActivation<T> Implements the Continuously Differentiable Exponential Linear Unit (CELU) activation function for neural networks.
ELUActivation<T> Implements the Exponential Linear Unit (ELU) activation function for neural networks.
GELUActivation<T> Implements the Gaussian Error Linear Unit (GELU) activation function for neural networks.
GaussianActivation<T> Implements the Gaussian activation function for neural networks.
GumbelSoftmaxActivation<T> Implements the Gumbel-Softmax activation function for neural networks, which enables differentiable sampling from discrete distributions.
HardSigmoidActivation<T> Implements the Hard Sigmoid activation function for neural networks.
HardSwishActivation<T> Implements the Hard Swish activation function used in MobileNetV3.
HardTanhActivation<T> Implements the Hard Tanh activation function for neural networks.
HierarchicalSoftmaxActivation<T> Implements the Hierarchical Softmax activation function for neural networks.
ISRUActivation<T> Implements the Inverse Square Root Unit (ISRU) activation function for neural networks.
IdentityActivation<T> Implements the Identity activation function for neural networks.
LeakyReLUActivation<T> Implements the Leaky Rectified Linear Unit (Leaky ReLU) activation function for neural networks.
LiSHTActivation<T> Implements the Linearly Scaled Hyperbolic Tangent (LiSHT) activation function for neural networks.
LogSoftmaxActivation<T> Implements the LogSoftmax activation function for neural networks.
LogSoftminActivation<T> Implements the LogSoftmin activation function for neural networks.
MaxoutActivation<T> Implements the Maxout activation function for neural networks.
MishActivation<T> Implements the Mish activation function for neural networks.
PReLUActivation<T> Implements the Parametric Rectified Linear Unit (PReLU) activation function for neural networks.
RReLUActivation<T> Implements the Randomized Rectified Linear Unit (RReLU) activation function.
ReLU6Activation<T> Implements the ReLU6 (Rectified Linear Unit capped at 6) activation function.
ReLUActivation<T> Implements the Rectified Linear Unit (ReLU) activation function.
SELUActivation<T> Implements the Scaled Exponential Linear Unit (SELU) activation function.
SQRBFActivation<T> Implements the Squared Radial Basis Function (SQRBF) activation function.
ScaledTanhActivation<T> Implements the Scaled Hyperbolic Tangent (tanh) activation function for neural networks.
SiLUActivation<T> Implements the SiLU (Sigmoid Linear Unit) activation function, also known as Swish.
SigmoidActivation<T> Implements the Sigmoid activation function, one of the most common activation functions in neural networks.
SignActivation<T> Implements the Sign activation function, which returns -1 for negative inputs, 1 for positive inputs, and 0 for zero.
SoftPlusActivation<T> Implements the SoftPlus activation function, which is a smooth approximation of the ReLU function.
SoftSignActivation<T> Implements the SoftSign activation function, which is a smooth alternative to the tanh function.
SoftmaxActivation<T> Implements the Softmax activation function, which converts a vector of real numbers into a probability distribution.
SoftminActivation<T> Implements the Softmin activation function, which is the opposite of Softmax and highlights the smallest values in a vector.
SparsemaxActivation<T> Implements the Sparsemax activation function, which is an alternative to Softmax that can produce sparse probability distributions.
SphericalSoftmaxActivation<T> Implements the Spherical Softmax activation function, which normalizes inputs to the unit sphere before applying softmax.
SquashActivation<T> Implements the Squash activation function, which normalizes vectors to have a magnitude between 0 and 1.
SwishActivation<T> Implements the Swish activation function for neural networks.
TanhActivation<T> Implements the Hyperbolic Tangent (tanh) activation function for neural networks.
TaylorSoftmaxActivation<T> Implements the Taylor Softmax activation function, which is a computationally efficient approximation of the standard Softmax function.
ThresholdedReLUActivation<T> Implements the Thresholded ReLU activation function, a variant of the standard ReLU function with an adjustable threshold.

Base Classes (1)

Type Summary
ActivationFunctionBase<T> Base class for all activation functions used in neural networks.

Interfaces (1)

Type Summary
IFusedActivation Implemented by scalar activation functions that have an exact fused-kernel equivalent (FusedActivationType), so fused inference paths such as IEngine.MlpForward can ask the activation what kernel it maps to instead of switching on its t…