MoEFeedForwardLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Sparse mixture-of-experts feed-forward sublayer (Mixtral / sparse-MoE): a linear router selects the top-k of E gated-SwiGLU experts per token and mixes their outputs by the renormalized router weights — y = Σ_{e∈topk(x)} softmax(router(x))_e · down_e(act(gate_e(x)) · up_e(x)).

Inheritance: LayerBase<T> → MoEFeedForwardLayer<T>

Implements: IShapeContract

How It Works

Inference-oriented (loaded from a pretrained checkpoint, not trained here): the routing is a non-differentiable top-k selection, so SupportsTraining is false and the forward pass gathers each expert’s routed tokens, runs that expert’s SwiGLU FFN, and scatters the weighted result back. Weights are the router plus, per expert, gate/up/down projections (all bias-free).

Constructors

Constructor Summary
MoEFeedForwardLayer(int hiddenSize, int ffnDim, int numExperts, int topK, IActivationFunction<T> activation, int sharedFfnDim) Creates a sparse MoE feed-forward layer.

Properties

Property Summary
DenseLayer<T> Router { get; } The router (gating) linear projection, hidden -> numExperts (bias-free).
DenseLayer<T> SharedDown { get; } The shared expert’s down projection (null when absent).
DenseLayer<T> SharedGate { get; } The shared expert’s gate projection (null when absent).
DenseLayer<T> SharedGateLogit { get; } The shared expert’s sigmoid gate (hidden -> 1; null when absent).
DenseLayer<T> SharedUp { get; } The shared expert’s up projection (null when absent).
bool HasSharedExpert { get; } Whether this layer has an always-on shared expert (Qwen2-MoE).
bool SupportsTraining { get; }
int NumExperts { get; } The number of experts.
int TopK { get; } The number of experts activated per token.

Methods

Method Summary
DenseLayer<T> ExpertDown(int e) The down projection of expert e.
DenseLayer<T> ExpertGate(int e) The gate projection of expert e.
DenseLayer<T> ExpertUp(int e) The up (value) projection of expert e.
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
void ClearGradients()
void Materialize() Forces lazy weight allocation for the router and every expert’s gate/up/down projections.
void ResetState()
void UpdateParameters(T learningRate)