PostNormTransformerDecoderBlock<T>
Layers · AiDotNet.NeuralNetworks.Layers
The decoder layer of the original Transformer (Vaswani et al. 2017, §3.1): masked multi-head self-attention, multi-head attention over the encoder output, and a position-wise feed-forward network, each followed by dropout, a residual connection and layer normalization (post-normalization).
Inheritance: LayerBase<T> → PostNormTransformerDecoderBlock<T>
Implements: IShapeContract
For Beginners
Each output position looks back at what has been produced so far, then at the encoded input, then refines itself — never peeking at positions that have not been produced yet.
How It Works
Self-attention is causal: position i attends only to positions ≤ i (“masking … ensures that
the predictions for position i can depend only on the known outputs at positions less than i”). Inputs: the
target sequence [time, hidden] or [batch, time, hidden] and the encoder memory of the same rank.
Transformer TTS (Li et al. 2019, §3.6) uses this decoder.
TransformerDecoderBlock is the pre-normalization variant; TransformerDecoderLayer
applies its activation to the attention outputs and has no dropout.
Constructors
| Constructor | Summary |
|---|---|
PostNormTransformerDecoderBlock(int hiddenSize, int numHeads, int ffnDim, double dropoutRate) |
Creates a decoder layer. |
Properties
| Property | Summary |
|---|---|
bool RequiresMultipleInputs { get; } |
|
bool SupportsTraining { get; } |
|
int HiddenSize { get; } |
Model width. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
Tensor<T> Forward(Tensor<T> target, Tensor<T> memory) |
Runs the layer on the target sequence with the encoder memory. |
void ResetState() |