TransformerDecoderBlock<T>
Layers · AiDotNet.NeuralNetworks.Layers
Pre-Layer-Normalization transformer decoder block — self-attention, a second (“cross”) attention, and a position-wise feed-forward network, each wrapped in a residual (skip) connection with layer normalization applied BEFORE the sublayer.
Inheritance: LayerBase<T> → TransformerDecoderBlock<T>
Implements: IShapeContract
How It Works
Block structure (Pre-LN; residual/FFN design Vaswani 2017 §3.1):
As with TransformerEncoderBlock, the residual connections
are what let the input signal flow through depth — their absence was the root
cause of issue #1380’s mode-collapse. This block restores them for the decoder
stack. NOTE: in this sequential model the “cross” attention attends over the
decoder’s own hidden stream (there is no separate encoder-memory input threaded
through the layer list); that pre-existing limitation is unchanged here — this
block only adds the missing residual connections and Pre-LN ordering.
Constructors
| Constructor | Summary |
|---|---|
TransformerDecoderBlock(int hiddenSize, int numHeads, int ffnDim, double dropoutRate, IActivationFunction<T> ffnActivation) |
Initialises a Pre-LN transformer decoder block. |
Properties
| Property | Summary |
|---|---|
LayerBase<T> FfnDownLayer { get; } |
The block’s current FFN down-projection (ffnDim → hiddenSize) sublayer. |
LayerBase<T> FfnUpLayer { get; } |
The block’s current FFN up-projection (hiddenSize → ffnDim) sublayer. |
LayerBase<T> SelfAttentionLayer { get; } |
The block’s current self-attention sublayer (replaceable via ReplaceSelfAttention). |
MultiHeadAttentionLayer<T> CrossAttentionLayer { get; } |
The block’s cross-attention sublayer. |
bool SupportsTraining { get; } |
|
double DropoutRate { get; } |
Dropout probability — persisted for deserialization. |
int FfnDim { get; } |
Feed-forward inner dimension — persisted for deserialization. |
int HiddenSize { get; } |
Model (feature) dimension — persisted for deserialization. |
int NumHeads { get; } |
Number of attention heads — persisted for deserialization. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
Tensor<T> Forward(Tensor<T> input, Tensor<T> encoderOutput) |
Pre-LN forward pass with a true encoder-decoder cross-attention sublayer: the cross-attention queries come from the decoder stream and the keys/values from encoderOutput (Vaswani 2017 §3.2.3). |
void ClearGradients() |
|
void ReplaceFfnDown(LayerBase<T> replacement) |
Swaps the FFN down-projection sublayer. |
void ReplaceFfnUp(LayerBase<T> replacement) |
Swaps the FFN up-projection sublayer. |
void ReplaceSelfAttention(LayerBase<T> replacement) |
Swaps the self-attention sublayer (e.g. |
void ResetState() |
|
void SetTrainingMode(bool isTraining) |
|
void UpdateParameters(T learningRate) |