MoEFeedForwardLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
Sparse mixture-of-experts feed-forward sublayer (Mixtral / sparse-MoE): a linear router selects the
top-k of E gated-SwiGLU experts per token and mixes their outputs by the renormalized
router weights — y = Σ_{e∈topk(x)} softmax(router(x))_e · down_e(act(gate_e(x)) · up_e(x)).
Inheritance: LayerBase<T> → MoEFeedForwardLayer<T>
Implements: IShapeContract
How It Works
Inference-oriented (loaded from a pretrained checkpoint, not trained here): the routing is a
non-differentiable top-k selection, so SupportsTraining is false and the forward
pass gathers each expert’s routed tokens, runs that expert’s SwiGLU FFN, and scatters the weighted
result back. Weights are the router plus, per expert, gate/up/down projections (all bias-free).
Constructors
| Constructor | Summary |
|---|---|
MoEFeedForwardLayer(int hiddenSize, int ffnDim, int numExperts, int topK, IActivationFunction<T> activation, int sharedFfnDim) |
Creates a sparse MoE feed-forward layer. |
Properties
| Property | Summary |
|---|---|
DenseLayer<T> Router { get; } |
The router (gating) linear projection, hidden -> numExperts (bias-free). |
DenseLayer<T> SharedDown { get; } |
The shared expert’s down projection (null when absent). |
DenseLayer<T> SharedGate { get; } |
The shared expert’s gate projection (null when absent). |
DenseLayer<T> SharedGateLogit { get; } |
The shared expert’s sigmoid gate (hidden -> 1; null when absent). |
DenseLayer<T> SharedUp { get; } |
The shared expert’s up projection (null when absent). |
bool HasSharedExpert { get; } |
Whether this layer has an always-on shared expert (Qwen2-MoE). |
bool SupportsTraining { get; } |
|
int NumExperts { get; } |
The number of experts. |
int TopK { get; } |
The number of experts activated per token. |
Methods
| Method | Summary |
|---|---|
DenseLayer<T> ExpertDown(int e) |
The down projection of expert e. |
DenseLayer<T> ExpertGate(int e) |
The gate projection of expert e. |
DenseLayer<T> ExpertUp(int e) |
The up (value) projection of expert e. |
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
void ClearGradients() |
|
void Materialize() |
Forces lazy weight allocation for the router and every expert’s gate/up/down projections. |
void ResetState() |
|
void UpdateParameters(T learningRate) |