MixtureOfExpertsGatingNoise

Enums · AiDotNet.Enums

How a MixtureOfExpertsLayer scales the Gaussian noise it adds to the gating logits during training.

For Beginners

keep the default to get the gating described in the paper. Choose SoftplusOfRouterLogits only to reproduce models trained with the earlier behaviour.

How It Works

Noisy top-k gating (Shazeer et al. 2017, eq. 4) is H(x) = x W_g + StandardNormal() * Softplus(x W_noise). The noise scale has its own learned weights, so the layer can learn how much to explore per input and per expert.

Fields

Field Summary
MixtureOfExpertsGatingNoise LearnedNoiseWeights Noise scaled by Softplus(x W_noise) with learned, zero-initialized W_noise (paper).
MixtureOfExpertsGatingNoise SoftplusOfRouterLogits Noise scaled by Softplus of the router’s own logits, the layer’s earlier behaviour.
int value__