GumbelSoftmaxActivation<T>

Models & Types · AiDotNet.ActivationFunctions

Implements the Gumbel-Softmax activation function for neural networks, which enables differentiable sampling from discrete distributions.

Inheritance: ActivationFunctionBase<T> → GumbelSoftmaxActivation<T>

For Beginners

The Gumbel-Softmax is a special activation function that helps neural networks make categorical (multiple-choice) decisions while still allowing for gradient-based learning.

Imagine you want your neural network to choose between several options (like choosing a word from a vocabulary). Normally, this would require a non-differentiable “hard” selection, which makes training difficult. Gumbel-Softmax solves this by:

  1. Adding randomness (Gumbel noise) to the inputs
  2. Using a “temperature” parameter to control how “certain” the choices are
  3. Producing a probability distribution that can approximate discrete choices

At high temperatures, the output is very “soft” (all options get some probability). At low temperatures, the output becomes more like a one-hot vector (one option gets almost all probability).

This technique is widely used in generative models, reinforcement learning, and any neural network that needs to make discrete choices while remaining differentiable.

Example

using AiDotNet.ActivationFunctions;

var activation = new GumbelSoftmaxActivation<double>();
double y = activation.Activate(0.5);
Console.WriteLine($"GumbelSoftmaxActivation: f(0.5) = {y:F4}");

Constructors

Constructor Summary
GumbelSoftmaxActivation(double temperature, int? seed) Initializes a new instance of the GumbelSoftmaxActivation class.

Methods

Method Summary
Matrix<T> Derivative(Vector<T> input) Calculates the derivative (Jacobian matrix) of the Gumbel-Softmax function for a vector of input values.
Tensor<T> Activate(Tensor<T> input) Applies Gumbel-Softmax to a tensor via engine primitives so the gradient tape records every step.
Vector<T> Activate(Vector<T> input) Applies Gumbel-Softmax to a tensor via engine primitives so the gradient tape records every step.