MixtureOfExpertsLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
Implements a Mixture-of-Experts (MoE) layer that routes inputs through multiple expert networks.
Inheritance: LayerBase<T> → MixtureOfExpertsLayer<T>
Implements: IAuxiliaryLossLayer<T>, IShapeContract
For Beginners
Think of a Mixture-of-Experts as a team of specialists working together.
How it works:
- You have multiple “experts” (specialized neural networks)
- A “router” (gating network) decides which experts should handle each input
- Each expert processes the input independently
- The final output is a weighted combination of the experts’ outputs
Why use MoE:
- Scalability: Add more experts to increase model capacity without proportionally increasing computation
- Specialization: Different experts learn to handle different types of inputs
- Efficiency: Only activate the most relevant experts for each input (sparse MoE)
Real-world analogy: Imagine you’re running a hospital with specialists:
- A cardiologist (expert 1) handles heart problems
- A neurologist (expert 2) handles brain issues
- A pediatrician (expert 3) handles children’s health
- A triage nurse (router) directs patients to the right specialist(s)
The router learns to send cardiac patients to the cardiologist, neurological cases to the neurologist, etc. This is more efficient than having one doctor handle everything, and allows each specialist to become highly skilled in their area.
How It Works
A Mixture-of-Experts layer contains multiple expert networks and a gating/routing network. For each input, the router determines how much weight to give each expert’s output, allowing the model to specialize different experts for different types of inputs. This architecture enables models with very high capacity while remaining computationally efficient by activating only a subset of parameters per input.
Key Features:
- Support for any number of experts
- Learned routing via a dense gating network
- Softmax routing: All experts contribute with learned weights
- Top-K routing: Only the top K experts are activated per input
- Load balancing: Optional auxiliary loss to encourage balanced expert usage
Constructors
| Constructor | Summary |
|---|---|
MixtureOfExpertsLayer(List<ILayer<T>> experts, ILayer<T> router, int[] inputShape, int[] outputShape, int topK, IActivationFunction<T> activationFunction, bool useLoadBalancing, T loadBalancingWeight, MixtureOfExpertsGatingNoise gatingNoise) |
Initializes a new instance of the MixtureOfExpertsLayer class. |
Properties
| Property | Summary |
|---|---|
T AuxiliaryLossWeight { get; set; } |
Gets or sets the weight for the auxiliary load balancing loss. |
bool SupportsTraining { get; } |
Gets a value indicating whether this layer supports training. |
bool UseAuxiliaryLoss { get; set; } |
Gets or sets a value indicating whether to use the auxiliary load balancing loss. |
int NumExperts { get; } |
Gets the number of experts in this MoE layer. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetAuxiliaryLossDiagnostics() |
Gets diagnostic information about expert usage and load balancing. |
Dictionary<string, string> GetDiagnostics() |
Gets diagnostic information about this component’s state and behavior. |
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
|
T ComputeAuxiliaryLoss() |
Computes the load balancing auxiliary loss based on expert usage from the last forward pass. |
void ClearGradients() |
|
void ResetState() |
Resets the internal state of the layer, clearing all cached values. |
void UpdateParameters(T learningRate) |
Updates all trainable parameters using the specified learning rate. |