TransformerDecoderBlock<T>

Layers · AiDotNet.NeuralNetworks.Layers

Pre-Layer-Normalization transformer decoder block — self-attention, a second (“cross”) attention, and a position-wise feed-forward network, each wrapped in a residual (skip) connection with layer normalization applied BEFORE the sublayer.

Inheritance: LayerBase<T> → TransformerDecoderBlock<T>

Implements: IShapeContract

How It Works

Block structure (Pre-LN; residual/FFN design Vaswani 2017 §3.1):

As with TransformerEncoderBlock, the residual connections are what let the input signal flow through depth — their absence was the root cause of issue #1380’s mode-collapse. This block restores them for the decoder stack. NOTE: in this sequential model the “cross” attention attends over the decoder’s own hidden stream (there is no separate encoder-memory input threaded through the layer list); that pre-existing limitation is unchanged here — this block only adds the missing residual connections and Pre-LN ordering.

Constructors

Constructor Summary
TransformerDecoderBlock(int hiddenSize, int numHeads, int ffnDim, double dropoutRate, IActivationFunction<T> ffnActivation) Initialises a Pre-LN transformer decoder block.

Properties

Property Summary
LayerBase<T> FfnDownLayer { get; } The block’s current FFN down-projection (ffnDim → hiddenSize) sublayer.
LayerBase<T> FfnUpLayer { get; } The block’s current FFN up-projection (hiddenSize → ffnDim) sublayer.
LayerBase<T> SelfAttentionLayer { get; } The block’s current self-attention sublayer (replaceable via ReplaceSelfAttention).
MultiHeadAttentionLayer<T> CrossAttentionLayer { get; } The block’s cross-attention sublayer.
bool SupportsTraining { get; }
double DropoutRate { get; } Dropout probability — persisted for deserialization.
int FfnDim { get; } Feed-forward inner dimension — persisted for deserialization.
int HiddenSize { get; } Model (feature) dimension — persisted for deserialization.
int NumHeads { get; } Number of attention heads — persisted for deserialization.

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
Tensor<T> Forward(Tensor<T> input, Tensor<T> encoderOutput) Pre-LN forward pass with a true encoder-decoder cross-attention sublayer: the cross-attention queries come from the decoder stream and the keys/values from encoderOutput (Vaswani 2017 §3.2.3).
void ClearGradients()
void ReplaceFfnDown(LayerBase<T> replacement) Swaps the FFN down-projection sublayer.
void ReplaceFfnUp(LayerBase<T> replacement) Swaps the FFN up-projection sublayer.
void ReplaceSelfAttention(LayerBase<T> replacement) Swaps the self-attention sublayer (e.g.
void ResetState()
void SetTrainingMode(bool isTraining)
void UpdateParameters(T learningRate)