HierarchicalSoftmaxActivation<T>

Models & Types · AiDotNet.ActivationFunctions

Implements the Hierarchical Softmax activation function for neural networks.

Inheritance: ActivationFunctionBase<T> → HierarchicalSoftmaxActivation<T>

For Beginners

Hierarchical Softmax is an efficient alternative to the standard Softmax function, especially when dealing with a large number of output classes (like thousands of words in language models).

While regular Softmax calculates probabilities for all possible classes at once (which can be slow with many classes), Hierarchical Softmax organizes classes in a tree structure:

  • Think of it like a “20 Questions” game where each question narrows down the possibilities
  • Each node in the tree represents a binary decision (left or right)
  • The final probability is calculated by multiplying probabilities along the path to a class

This approach reduces computation from O(N) to O(log N), where N is the number of classes, making it much faster for problems with many output classes.

Common uses include:

  • Natural language processing (predicting words from vocabularies)
  • Classification problems with many categories
  • Any task where computing standard Softmax would be too slow

Constructors

Constructor Summary
HierarchicalSoftmaxActivation(int numClasses) Initializes a new instance of the Hierarchical Softmax activation function.

Properties

Property Summary
Tensor<T> NodeWeightsTensor { get; } Gets the node weights as a tensor.

Methods

Method Summary
Matrix<T> Derivative(Vector<T> input) Calculates the derivative (gradient) of the Hierarchical Softmax function.
Tensor<T> Activate(Tensor<T> input) Applies HierarchicalSoftmax to a tensor via engine primitives so the gradient tape records every step.
Vector<T> Activate(Vector<T> input) Applies HierarchicalSoftmax to a tensor via engine primitives so the gradient tape records every step.