ExpertLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Represents an expert module in a Mixture-of-Experts architecture, containing a sequence of layers.

Inheritance: LayerBase<T> → ExpertLayer<T>

Implements: IShapeContract

For Beginners

Think of an Expert as a mini neural network that specializes in a particular task.

In a Mixture-of-Experts system:

  • You have multiple “experts” (mini-networks), each with their own layers
  • Each expert learns to be good at handling certain types of inputs
  • A routing mechanism decides which experts should process each input
  • The final output combines the predictions from the selected experts

For example, in a language model:

  • One expert might specialize in technical vocabulary
  • Another might handle conversational language
  • Another might focus on formal writing
  • The router learns to send each input to the most appropriate expert(s)

This allows the model to scale to very large sizes while keeping computation efficient, since only a subset of experts are activated for each input.

How It Works

An Expert is a container for a sequence of neural network layers that are executed sequentially. In a Mixture-of-Experts (MoE) architecture, multiple experts process the same input, and their outputs are combined based on learned routing weights. Each expert can specialize in processing different types of inputs or patterns.

Constructors

Constructor Summary
ExpertLayer(List<ILayer<T>> layers, int[] inputShape, int[] outputShape, IActivationFunction<T> activationFunction) Initializes a new instance of the ExpertLayer class with the specified layers.

Properties

Property Summary
bool SupportsTraining { get; } Gets a value indicating whether this expert supports training through backpropagation.

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
void ClearGradients()
void ResetState() Resets the internal state of all layers, clearing any cached values.
void UpdateParameters(T learningRate) Updates all trainable parameters in all layers using the specified learning rate.