MixtureOfExpertsGatingNoise
Enums · AiDotNet.Enums
How a MixtureOfExpertsLayer scales the Gaussian noise
it adds to the gating logits during training.
For Beginners
keep the default to get the gating described in the paper. Choose
SoftplusOfRouterLogits only to reproduce models trained with the earlier behaviour.
How It Works
Noisy top-k gating (Shazeer et al. 2017, eq. 4) is
H(x) = x W_g + StandardNormal() * Softplus(x W_noise). The noise scale has its own learned
weights, so the layer can learn how much to explore per input and per expert.
Fields
| Field | Summary |
|---|---|
MixtureOfExpertsGatingNoise LearnedNoiseWeights |
Noise scaled by Softplus(x W_noise) with learned, zero-initialized W_noise (paper). |
MixtureOfExpertsGatingNoise SoftplusOfRouterLogits |
Noise scaled by Softplus of the router’s own logits, the layer’s earlier behaviour. |
int value__ |