MixtureOfExpertsOptions<T>
Options & Configuration · AiDotNet.Models.Options
Configuration options for the Mixture-of-Experts (MoE) neural network model.
Inheritance: ModelOptions → NeuralNetworkOptions → ModelHyperparameterOptions → MixtureOfExpertsOptions<T>
For Beginners
Mixture-of-Experts (MoE) is like having a team of specialists rather than one generalist.
Imagine you’re running a hospital:
- Instead of one doctor handling everything, you have specialists (cardiologist, neurologist, etc.)
- A triage system (gating network) decides which specialist(s) should see each patient
- Each specialist only handles cases they’re best suited for
In a MoE neural network:
- Multiple “expert” networks specialize in different patterns in your data
- A “gating network” learns to route each input to the best expert(s)
- Only a few experts process each input (sparse activation), making it efficient
- The final prediction combines the outputs from the selected experts
This class lets you configure:
- How many expert networks to use
- How many experts process each input (Top-K)
- Dimensions of the expert networks
- Whether to use load balancing to ensure all experts are utilized
How It Works
Mixture-of-Experts is a neural network architecture that employs multiple specialist networks (experts) and a gating mechanism to route inputs to the most appropriate experts. This approach enables:
- Increased model capacity without proportional compute cost (sparse activation)
- Specialization of different experts on different aspects of the problem
- Improved scalability for large-scale problems
Constructors
| Constructor | Summary |
|---|---|
MixtureOfExpertsOptions() |
Properties
| Property | Summary |
|---|---|
bool UseLoadBalancing { get; set; } |
Gets or sets whether to enable auxiliary load balancing loss. |
double LoadBalancingWeight { get; set; } |
Gets or sets the weight of the auxiliary load balancing loss. |
int HiddenExpansion { get; set; } |
Gets or sets the hidden layer expansion factor for each expert’s feed-forward network. |
int InputDim { get; set; } |
Gets or sets the input dimension for each expert network. |
int NumExperts { get; set; } |
Gets or sets the number of expert networks in the mixture. |
int OutputDim { get; set; } |
Gets or sets the output dimension for each expert network. |
int TopK { get; set; } |
Gets or sets the number of experts to activate for each input (Top-K routing). |
Methods
| Method | Summary |
|---|---|
void Validate() |
Validates that all option values are within acceptable ranges. |