MixtureOfExpertsNeuralNetwork<T>
Models & Types · AiDotNet.NeuralNetworks
Represents a Mixture-of-Experts (MoE) neural network that routes inputs through multiple specialist networks.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → VectorModelLayoutBase<T> → MixtureOfExpertsNeuralNetwork<T>
For Beginners
Mixture-of-Experts is like having a team of specialists rather than one generalist.
Imagine you’re running a hospital:
- Instead of one doctor handling everything, you have specialists (cardiologist, neurologist, etc.)
- A triage system (gating network) decides which specialist(s) should see each patient
- Each specialist only handles cases they’re best suited for
In a MoE neural network:
- Multiple “expert” networks specialize in different patterns in your data
- A “gating network” learns to route each input to the best expert(s)
- Only a few experts process each input (sparse activation), making it efficient
- The final prediction combines the outputs from the selected experts
This model automatically implements IFullModel, allowing it to work with AiModelBuilder just like any other neural network in AiDotNet.
How It Works
A Mixture-of-Experts neural network employs multiple expert networks and a gating mechanism to route inputs to the most appropriate experts. This architecture enables:
- Increased model capacity without proportional compute cost (sparse activation)
- Specialization of different experts on different aspects of the problem
- Improved scalability for large-scale problems
The architecture consists of:
- Multiple expert networks (can be feed-forward, convolutional, etc.)
- A gating/routing network that learns to select appropriate experts
- Optional load balancing loss to ensure all experts are utilized
Key Features:
- Configurable number of expert networks
- Top-K sparse routing for computational efficiency
- Automatic load balancing to prevent expert collapse
- Integration with AiModelBuilder for easy training
- Full support for serialization and deserialization
Example
var options = new MixtureOfExpertsOptions<double> { InputDim = 64, NumExperts = 8, TopK = 2};
var input = Tensor<float>.CreateRandom(new[] { 1, 64 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new MixtureOfExpertsNeuralNetwork<float>())
.Build(trainX, trainY);
var output = result.Predict(input);
Constructors
| Constructor | Summary |
|---|---|
MixtureOfExpertsNeuralNetwork() |
Initializes a new instance of the MixtureOfExpertsNeuralNetwork class. |
MixtureOfExpertsNeuralNetwork(MixtureOfExpertsOptions<T> options, NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Indicates whether this network supports training. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
Retrieves metadata about the Mixture-of-Experts neural network model. |
Tensor<T> Forward(Tensor<T> input) |
Performs a forward pass through the network with the given input tensor. |