PostNormTransformerDecoderBlock<T>

Layers · AiDotNet.NeuralNetworks.Layers

The decoder layer of the original Transformer (Vaswani et al. 2017, §3.1): masked multi-head self-attention, multi-head attention over the encoder output, and a position-wise feed-forward network, each followed by dropout, a residual connection and layer normalization (post-normalization).

Inheritance: LayerBase<T> → PostNormTransformerDecoderBlock<T>

Implements: IShapeContract

For Beginners

Each output position looks back at what has been produced so far, then at the encoded input, then refines itself — never peeking at positions that have not been produced yet.

How It Works

Self-attention is causal: position i attends only to positions ≤ i (“masking … ensures that the predictions for position i can depend only on the known outputs at positions less than i”). Inputs: the target sequence [time, hidden] or [batch, time, hidden] and the encoder memory of the same rank. Transformer TTS (Li et al. 2019, §3.6) uses this decoder.

TransformerDecoderBlock is the pre-normalization variant; TransformerDecoderLayer applies its activation to the attention outputs and has no dropout.

Constructors

Constructor Summary
PostNormTransformerDecoderBlock(int hiddenSize, int numHeads, int ffnDim, double dropoutRate) Creates a decoder layer.

Properties

Property Summary
bool RequiresMultipleInputs { get; }
bool SupportsTraining { get; }
int HiddenSize { get; } Model width.

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
Tensor<T> Forward(Tensor<T> target, Tensor<T> memory) Runs the layer on the target sequence with the encoder memory.
void ResetState()