Activation Functions

40

Every activation function from classic ReLU to modern gating mechanisms

40 activation functions covering all standard (ReLU, GELU, Sigmoid), gated (SwiGLU, GeGLU, GLU), and specialized (Hardswish, CELU, Mish) variants. All SIMD-vectorized for maximum throughput.

Neural Network Design LLM Architecture Efficient Inference Mobile AI Custom Architectures

Standard Activations

Foundational activation functions used across all neural network types.

ReLU

Rectified Linear Unit - the default activation for most architectures.

GELU

Gaussian Error Linear Unit used in BERT and GPT architectures.

SiLU / Swish

Sigmoid Linear Unit used in EfficientNet and modern architectures.

Mish

Self-regularized non-monotonic activation used in YOLOv4.

Sigmoid

Logistic function for binary outputs and gates.

Softmax

Normalized exponential for probability distributions.

Gated Activations

Gating mechanisms used in modern LLMs and efficient architectures.

SwiGLU

Swish-Gated Linear Unit used in LLaMA and modern LLMs.

GeGLU

GELU-Gated Linear Unit for transformer feed-forward networks.

GLU

Gated Linear Unit with learnable gates.

ReGLU

ReLU-Gated Linear Unit for efficient computation.

Maxout

Piecewise linear activation with multiple feature maps.

Specialized Activations

Task-specific activations for mobile, quantization, and specialized needs.

Hardswish

Hardware-efficient approximation of Swish for mobile deployment.

LeakyReLU / PReLU

Non-zero gradient for negative inputs preventing dead neurons.

ELU / SELU

Exponential linear units with self-normalizing properties.

CELU

Continuously differentiable ELU for smoother gradients.

Activations with AiModelBuilder

C# 
using AiDotNet;

// Build a model with custom activations using AiModelBuilder
var result = await new AiModelBuilder<float, float[], float>()
    .ConfigureModel(new NeuralNetwork<float>(
        inputSize: 768, hiddenSize: 256, outputSize: 10,
        activation: new SwiGLU<float>()))
    .ConfigureOptimizer(new AdamOptimizer<float>())
    .ConfigurePreprocessing()
    .BuildAsync(features, labels);

var prediction = result.Predict(newSample);

Start building with Activation Functions

All 40 implementations are included free under Apache 2.0.

Install AiDotNet Browse All Features