DbrxDecoderBlock<T>
Layers · AiDotNet.NeuralNetworks.Layers
DBRX pre-LN decoder block: y = x + Attn(LayerNorm(x)) then z = y + MoE(LayerNorm(y)). Like
MoEDecoderBlock but with true (mean-centered) LayerNorms, matching DBRX’s
norm_attn_norm structure.
Inheritance: LayerBase<T> → DbrxDecoderBlock<T>
Implements: IShapeContract
Constructors
| Constructor | Summary |
|---|---|
DbrxDecoderBlock(int hiddenSize, LayerBase<T> attention, MoEFeedForwardLayer<T> moe, double layerNormEpsilon) |
Creates a DBRX LayerNorm MoE decoder block. |
Properties
| Property | Summary |
|---|---|
LayerBase<T> AttentionLayer { get; } |
The self-attention sublayer. |
LayerNormalizationLayer<T> Norm1 { get; } |
The pre-attention LayerNorm. |
LayerNormalizationLayer<T> Norm2 { get; } |
The pre-MoE LayerNorm. |
MoEFeedForwardLayer<T> Moe { get; } |
The mixture-of-experts feed-forward sublayer. |
bool SupportsTraining { get; } |
|
int HiddenSize { get; } |
The model (input/output) feature dimension. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
void ClearGradients() |
|
void ResetState() |
|
void UpdateParameters(T learningRate) |