MixtureOfExpertsNeuralNetwork<T>

Models & Types · AiDotNet.NeuralNetworks

Represents a Mixture-of-Experts (MoE) neural network that routes inputs through multiple specialist networks.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → VectorModelLayoutBase<T> → MixtureOfExpertsNeuralNetwork<T>

For Beginners

Mixture-of-Experts is like having a team of specialists rather than one generalist.

Imagine you’re running a hospital:

  • Instead of one doctor handling everything, you have specialists (cardiologist, neurologist, etc.)
  • A triage system (gating network) decides which specialist(s) should see each patient
  • Each specialist only handles cases they’re best suited for

In a MoE neural network:

  • Multiple “expert” networks specialize in different patterns in your data
  • A “gating network” learns to route each input to the best expert(s)
  • Only a few experts process each input (sparse activation), making it efficient
  • The final prediction combines the outputs from the selected experts

This model automatically implements IFullModel, allowing it to work with AiModelBuilder just like any other neural network in AiDotNet.

How It Works

A Mixture-of-Experts neural network employs multiple expert networks and a gating mechanism to route inputs to the most appropriate experts. This architecture enables:

  • Increased model capacity without proportional compute cost (sparse activation)
  • Specialization of different experts on different aspects of the problem
  • Improved scalability for large-scale problems

The architecture consists of:

  • Multiple expert networks (can be feed-forward, convolutional, etc.)
  • A gating/routing network that learns to select appropriate experts
  • Optional load balancing loss to ensure all experts are utilized

Key Features:

  • Configurable number of expert networks
  • Top-K sparse routing for computational efficiency
  • Automatic load balancing to prevent expert collapse
  • Integration with AiModelBuilder for easy training
  • Full support for serialization and deserialization

Example

var options = new MixtureOfExpertsOptions<double> { InputDim = 64, NumExperts = 8, TopK = 2};
var input = Tensor<float>.CreateRandom(new[] { 1, 64 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new MixtureOfExpertsNeuralNetwork<float>())
    .Build(trainX, trainY);
var output = result.Predict(input);

Constructors

Constructor Summary
MixtureOfExpertsNeuralNetwork() Initializes a new instance of the MixtureOfExpertsNeuralNetwork class.
MixtureOfExpertsNeuralNetwork(MixtureOfExpertsOptions<T> options, NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction)

Properties

Property Summary
bool SupportsTraining { get; } Indicates whether this network supports training.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata() Retrieves metadata about the Mixture-of-Experts neural network model.
Tensor<T> Forward(Tensor<T> input) Performs a forward pass through the network with the given input tensor.