DbrxDecoderBlock<T>

Layers · AiDotNet.NeuralNetworks.Layers

DBRX pre-LN decoder block: y = x + Attn(LayerNorm(x)) then z = y + MoE(LayerNorm(y)). Like MoEDecoderBlock but with true (mean-centered) LayerNorms, matching DBRX’s norm_attn_norm structure.

Inheritance: LayerBase<T> → DbrxDecoderBlock<T>

Implements: IShapeContract

Constructors

Constructor Summary
DbrxDecoderBlock(int hiddenSize, LayerBase<T> attention, MoEFeedForwardLayer<T> moe, double layerNormEpsilon) Creates a DBRX LayerNorm MoE decoder block.

Properties

Property Summary
LayerBase<T> AttentionLayer { get; } The self-attention sublayer.
LayerNormalizationLayer<T> Norm1 { get; } The pre-attention LayerNorm.
LayerNormalizationLayer<T> Norm2 { get; } The pre-MoE LayerNorm.
MoEFeedForwardLayer<T> Moe { get; } The mixture-of-experts feed-forward sublayer.
bool SupportsTraining { get; }
int HiddenSize { get; } The model (input/output) feature dimension.

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
void ClearGradients()
void ResetState()
void UpdateParameters(T learningRate)