ActivationFunctionBase<T>

Base Classes · AiDotNet.ActivationFunctions

Base class for all activation functions used in neural networks.

Implements: IActivationFunction<T>, IVectorActivationFunction<T>

Known derived types: BentIdentityActivation<T>, BinarySpikingActivation<T>, CELUActivation<T>, ELUActivation<T>, EntmoidActivation<T>, GELUActivation<T>, GaussianActivation<T>, GumbelSoftmaxActivation<T>, HardSigmoidActivation<T>, HardSwishActivation<T>, HardTanhActivation<T>, HierarchicalSoftmaxActivation<T>, ISRUActivation<T>, IdentityActivation<T>, LeakyReLUActivation<T>, LiSHTActivation<T>, LogSoftmaxActivation<T>, LogSoftminActivation<T>, MaxoutActivation<T>, MishActivation<T>, PReLUActivation<T>, RReLUActivation<T>, ReLU6Activation<T>, ReLUActivation<T>, SELUActivation<T>, SQRBFActivation<T>, ScaledTanhActivation<T>, SiLUActivation<T>, SigmoidActivation<T>, SignActivation<T>, SoftPlusActivation<T>, SoftSignActivation<T>, SoftmaxActivation<T>, SoftminActivation<T>, SparsemaxActivation<T>, SphericalSoftmaxActivation<T>, SquashActivation<T>, SwishActivation<T>, TanhActivation<T>, TaylorSoftmaxActivation<T>, ThresholdedReLUActivation<T>

For Beginners

Activation functions are mathematical operations that determine the output of a neural network node. They introduce non-linearity into the network, allowing it to learn complex patterns. Think of them as decision-makers that determine how strongly a neuron “fires” based on its inputs.

Common activation functions include:

  • Sigmoid: Outputs values between 0 and 1 (like probabilities)
  • ReLU: Returns 0 for negative inputs, or the input value for positive inputs
  • Tanh: Similar to sigmoid but outputs values between -1 and 1

The “derivative” methods are used during training to determine how to adjust the network’s weights to improve its accuracy.

Properties

Property Summary
bool SupportsGpuTraining { get; } Gets whether this activation function supports GPU-resident training.

Methods

Method Summary
Matrix<T> Derivative(Vector<T> input) Calculates the derivative for each element in a tensor.
T Activate(T input) Applies the activation function to each element in a tensor.
T Derivative(T input) Calculates the derivative for each element in a tensor.
Tensor<T> Activate(Tensor<T> input) Applies the activation function to each element in a tensor.
Tensor<T> Backward(Tensor<T> input, Tensor<T> outputGradient) Calculates the backward pass gradient for this activation function.
Tensor<T> Derivative(Tensor<T> input) Calculates the derivative for each element in a tensor.
Vector<T> Activate(Vector<T> input) Applies the activation function to each element in a tensor.
void ForwardGpu(IDirectGpuBackend backend, IGpuBuffer input, IGpuBuffer output, int size) Applies the activation function on GPU.