GELUActivation<T>

Models & Types · AiDotNet.ActivationFunctions

Implements the Gaussian Error Linear Unit (GELU) activation function for neural networks.

Inheritance: ActivationFunctionBase<T> → GELUActivation<T>

Implements: IFusedActivation

For Beginners

The GELU activation function is a modern activation function used in many state-of-the-art neural networks, including transformers like BERT and GPT.

Think of GELU as a “smoother” version of ReLU that:

  • Keeps most positive values (like ReLU does)
  • Gradually reduces small positive values
  • Gradually allows some small negative values through
  • Blocks large negative values (like ReLU does)

GELU can be thought of as multiplying the input by the probability that the input is positive. This creates a smooth curve that transitions naturally between allowing and blocking values, rather than having a sharp cutoff like ReLU.

GELU is widely used in modern language models and has been shown to perform better than older activation functions in many deep learning tasks.

Example

using AiDotNet.ActivationFunctions;

var activation = new GELUActivation<double>();
double y = activation.Activate(0.5);
Console.WriteLine($"GELUActivation: f(0.5) = {y:F4}");

Constructors

Constructor Summary
GELUActivation()

Properties

Property Summary
bool SupportsGpuTraining { get; } Gets whether GELU supports GPU-resident training.

Methods

Method Summary
T Activate(T input) Applies the GELU activation function to each element in a tensor.
T Derivative(T input) Calculates the derivative of the GELU function for each element in a tensor.
Tensor<T> Activate(Tensor<T> input) Applies the GELU activation function to each element in a tensor.
Tensor<T> Derivative(Tensor<T> input) Calculates the derivative of the GELU function for each element in a tensor.
void ForwardGpu(IDirectGpuBackend backend, IGpuBuffer input, IGpuBuffer output, int size) Applies the GELU activation function on GPU.