GELUActivation<T>
Models & Types · AiDotNet.ActivationFunctions
Implements the Gaussian Error Linear Unit (GELU) activation function for neural networks.
Inheritance: ActivationFunctionBase<T> → GELUActivation<T>
Implements: IFusedActivation
For Beginners
The GELU activation function is a modern activation function used in many state-of-the-art neural networks, including transformers like BERT and GPT.
Think of GELU as a “smoother” version of ReLU that:
- Keeps most positive values (like ReLU does)
- Gradually reduces small positive values
- Gradually allows some small negative values through
- Blocks large negative values (like ReLU does)
GELU can be thought of as multiplying the input by the probability that the input is positive. This creates a smooth curve that transitions naturally between allowing and blocking values, rather than having a sharp cutoff like ReLU.
GELU is widely used in modern language models and has been shown to perform better than older activation functions in many deep learning tasks.
Example
using AiDotNet.ActivationFunctions;
var activation = new GELUActivation<double>();
double y = activation.Activate(0.5);
Console.WriteLine($"GELUActivation: f(0.5) = {y:F4}");
Constructors
| Constructor | Summary |
|---|---|
GELUActivation() |
Properties
| Property | Summary |
|---|---|
bool SupportsGpuTraining { get; } |
Gets whether GELU supports GPU-resident training. |
Methods
| Method | Summary |
|---|---|
T Activate(T input) |
Applies the GELU activation function to each element in a tensor. |
T Derivative(T input) |
Calculates the derivative of the GELU function for each element in a tensor. |
Tensor<T> Activate(Tensor<T> input) |
Applies the GELU activation function to each element in a tensor. |
Tensor<T> Derivative(Tensor<T> input) |
Calculates the derivative of the GELU function for each element in a tensor. |
void ForwardGpu(IDirectGpuBackend backend, IGpuBuffer input, IGpuBuffer output, int size) |
Applies the GELU activation function on GPU. |