TransformerDecoderLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
Represents a transformer decoder layer that processes sequences using self-attention, cross-attention, and feed-forward networks.
Inheritance: LayerBase<T> → TransformerDecoderLayer<T>
Implements: IAuxiliaryLossLayer<T>, IShapeContract
For Beginners
This layer helps the network generate sequences while considering both what it has generated so far and input from another source.
Think of it like a writer who is translating a book:
- First, the writer looks at what they’ve translated so far to maintain consistency (self-attention)
- Then they look at the original text to understand what to translate next (cross-attention)
- Finally, they process all this information to produce the next part of the translation (feed-forward network)
For example, in machine translation, the decoder generates each word of the target language by:
- Looking at the words it has already generated (to maintain grammatical coherence)
- Looking at the encoded source sentence (to understand what content to translate)
- Combining this information to produce the most appropriate next word
This architecture is powerful for tasks like translation, summarization, and text generation.
How It Works
A transformer decoder layer is a fundamental building block of transformer-based models for sequence-to-sequence tasks. It consists of three main components: a masked self-attention mechanism that processes the target sequence, a cross-attention mechanism that attends to the encoder’s output, and a feed-forward network for additional transformation. Each component is followed by layer normalization and residual connections to facilitate training of deep networks.
Constructors
| Constructor | Summary |
|---|---|
TransformerDecoderLayer(int numHeads, int feedForwardDim, int sequenceLength, IActivationFunction<T> ffnActivation, double dropoutRate, bool causal) |
Lazy ctor: _embeddingSize is resolved from input.Shape[^1] on first Forward; the inner attention/cross-attention/ FFN/norm sublayers are constructed then. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<LayerPort> InputPorts { get; } |
Declares named input ports for this multi-input layer. |
T AuxiliaryLossWeight { get; set; } |
Gets or sets the weight for the auxiliary loss contribution. |
bool IsCausal { get; } |
Whether the self-attention is causal. |
bool SupportsTraining { get; } |
Gets a value indicating whether this layer supports training. |
bool UseAuxiliaryLoss { get; set; } |
Gets or sets a value indicating whether auxiliary loss is enabled for this layer. |
double DropoutRate { get; } |
Dropout probability on each sublayer output. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetAuxiliaryLossDiagnostics() |
Gets diagnostic information about the auxiliary loss computation. |
Dictionary<string, string> GetDiagnostics() |
Gets diagnostic information about this component’s state and behavior. |
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
T ComputeAuxiliaryLoss() |
Computes the auxiliary loss for this layer by aggregating losses from sublayers. |
Tensor<T> Forward(Tensor<T> input, Tensor<T> encoderOutput) |
Performs the forward pass of the transformer decoder layer. |
void ClearGradients() |
|
void ResetState() |
Resets the internal state of the transformer decoder layer and all its sublayers. |
void UpdateParameters(T learningRate) |
Updates the parameters of all sublayers using the calculated gradients. |
void UpdateParametersGpu(IGpuOptimizerConfig config) |
Updates layer parameters using GPU-resident optimizer. |