MoEDecoderBlock<T>
Layers · AiDotNet.NeuralNetworks.Layers
Pre-LN decoder block whose feed-forward sublayer is a sparse mixture-of-experts (Mixtral-style):
y = x + Attn(RMSNorm(x)), then z = y + MoE(RMSNorm(y)). Mirrors
PreLNTransformerBlock but routes each token through the top-k of many gated experts.
Inheritance: LayerBase<T> → MoEDecoderBlock<T>
Implements: IShapeContract
Constructors
| Constructor | Summary |
|---|---|
MoEDecoderBlock(int hiddenSize, LayerBase<T> attention, MoEFeedForwardLayer<T> moe, double rmsNormEpsilon) |
Creates a MoE pre-LN decoder block. |
Properties
| Property | Summary |
|---|---|
LayerBase<T> AttentionLayer { get; } |
The self-attention sublayer. |
MoEFeedForwardLayer<T> Moe { get; } |
The mixture-of-experts feed-forward sublayer. |
RMSNormalizationLayer<T> Norm1 { get; } |
The pre-attention RMSNorm. |
RMSNormalizationLayer<T> Norm2 { get; } |
The pre-MoE RMSNorm. |
bool SupportsTraining { get; } |
|
int HiddenSize { get; } |
The model (input/output) feature dimension. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
void ClearGradients() |
|
void ResetState() |
|
void UpdateParameters(T learningRate) |