MoEDecoderBlock<T>

Layers · AiDotNet.NeuralNetworks.Layers

Pre-LN decoder block whose feed-forward sublayer is a sparse mixture-of-experts (Mixtral-style): y = x + Attn(RMSNorm(x)), then z = y + MoE(RMSNorm(y)). Mirrors PreLNTransformerBlock but routes each token through the top-k of many gated experts.

Inheritance: LayerBase<T> → MoEDecoderBlock<T>

Implements: IShapeContract

Constructors

Constructor Summary
MoEDecoderBlock(int hiddenSize, LayerBase<T> attention, MoEFeedForwardLayer<T> moe, double rmsNormEpsilon) Creates a MoE pre-LN decoder block.

Properties

Property Summary
LayerBase<T> AttentionLayer { get; } The self-attention sublayer.
MoEFeedForwardLayer<T> Moe { get; } The mixture-of-experts feed-forward sublayer.
RMSNormalizationLayer<T> Norm1 { get; } The pre-attention RMSNorm.
RMSNormalizationLayer<T> Norm2 { get; } The pre-MoE RMSNorm.
bool SupportsTraining { get; }
int HiddenSize { get; } The model (input/output) feature dimension.

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
void ClearGradients()
void ResetState()
void UpdateParameters(T learningRate)