ActivationFunctionBase<T>
Base Classes · AiDotNet.ActivationFunctions
Base class for all activation functions used in neural networks.
Implements: IActivationFunction<T>, IVectorActivationFunction<T>
Known derived types: BentIdentityActivation<T>, BinarySpikingActivation<T>, CELUActivation<T>, ELUActivation<T>, EntmoidActivation<T>, GELUActivation<T>, GaussianActivation<T>, GumbelSoftmaxActivation<T>, HardSigmoidActivation<T>, HardSwishActivation<T>, HardTanhActivation<T>, HierarchicalSoftmaxActivation<T>, ISRUActivation<T>, IdentityActivation<T>, LeakyReLUActivation<T>, LiSHTActivation<T>, LogSoftmaxActivation<T>, LogSoftminActivation<T>, MaxoutActivation<T>, MishActivation<T>, PReLUActivation<T>, RReLUActivation<T>, ReLU6Activation<T>, ReLUActivation<T>, SELUActivation<T>, SQRBFActivation<T>, ScaledTanhActivation<T>, SiLUActivation<T>, SigmoidActivation<T>, SignActivation<T>, SoftPlusActivation<T>, SoftSignActivation<T>, SoftmaxActivation<T>, SoftminActivation<T>, SparsemaxActivation<T>, SphericalSoftmaxActivation<T>, SquashActivation<T>, SwishActivation<T>, TanhActivation<T>, TaylorSoftmaxActivation<T>, ThresholdedReLUActivation<T>
For Beginners
Activation functions are mathematical operations that determine the output of a neural network node. They introduce non-linearity into the network, allowing it to learn complex patterns. Think of them as decision-makers that determine how strongly a neuron “fires” based on its inputs.
Common activation functions include:
- Sigmoid: Outputs values between 0 and 1 (like probabilities)
- ReLU: Returns 0 for negative inputs, or the input value for positive inputs
- Tanh: Similar to sigmoid but outputs values between -1 and 1
The “derivative” methods are used during training to determine how to adjust the network’s weights to improve its accuracy.
Properties
| Property | Summary |
|---|---|
bool SupportsGpuTraining { get; } |
Gets whether this activation function supports GPU-resident training. |
Methods
| Method | Summary |
|---|---|
Matrix<T> Derivative(Vector<T> input) |
Calculates the derivative for each element in a tensor. |
T Activate(T input) |
Applies the activation function to each element in a tensor. |
T Derivative(T input) |
Calculates the derivative for each element in a tensor. |
Tensor<T> Activate(Tensor<T> input) |
Applies the activation function to each element in a tensor. |
Tensor<T> Backward(Tensor<T> input, Tensor<T> outputGradient) |
Calculates the backward pass gradient for this activation function. |
Tensor<T> Derivative(Tensor<T> input) |
Calculates the derivative for each element in a tensor. |
Vector<T> Activate(Vector<T> input) |
Applies the activation function to each element in a tensor. |
void ForwardGpu(IDirectGpuBackend backend, IGpuBuffer input, IGpuBuffer output, int size) |
Applies the activation function on GPU. |