GumbelSoftmaxActivation<T>
Models & Types · AiDotNet.ActivationFunctions
Implements the Gumbel-Softmax activation function for neural networks, which enables differentiable sampling from discrete distributions.
Inheritance: ActivationFunctionBase<T> → GumbelSoftmaxActivation<T>
For Beginners
The Gumbel-Softmax is a special activation function that helps neural networks make categorical (multiple-choice) decisions while still allowing for gradient-based learning.
Imagine you want your neural network to choose between several options (like choosing a word from a vocabulary). Normally, this would require a non-differentiable “hard” selection, which makes training difficult. Gumbel-Softmax solves this by:
- Adding randomness (Gumbel noise) to the inputs
- Using a “temperature” parameter to control how “certain” the choices are
- Producing a probability distribution that can approximate discrete choices
At high temperatures, the output is very “soft” (all options get some probability). At low temperatures, the output becomes more like a one-hot vector (one option gets almost all probability).
This technique is widely used in generative models, reinforcement learning, and any neural network that needs to make discrete choices while remaining differentiable.
Example
using AiDotNet.ActivationFunctions;
var activation = new GumbelSoftmaxActivation<double>();
double y = activation.Activate(0.5);
Console.WriteLine($"GumbelSoftmaxActivation: f(0.5) = {y:F4}");
Constructors
| Constructor | Summary |
|---|---|
GumbelSoftmaxActivation(double temperature, int? seed) |
Initializes a new instance of the GumbelSoftmaxActivation class. |
Methods
| Method | Summary |
|---|---|
Matrix<T> Derivative(Vector<T> input) |
Calculates the derivative (Jacobian matrix) of the Gumbel-Softmax function for a vector of input values. |
Tensor<T> Activate(Tensor<T> input) |
Applies Gumbel-Softmax to a tensor via engine primitives so the gradient tape records every step. |
Vector<T> Activate(Vector<T> input) |
Applies Gumbel-Softmax to a tensor via engine primitives so the gradient tape records every step. |