TransformerDecoderLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Represents a transformer decoder layer that processes sequences using self-attention, cross-attention, and feed-forward networks.

Inheritance: LayerBase<T> → TransformerDecoderLayer<T>

Implements: IAuxiliaryLossLayer<T>, IShapeContract

For Beginners

This layer helps the network generate sequences while considering both what it has generated so far and input from another source.

Think of it like a writer who is translating a book:

  • First, the writer looks at what they’ve translated so far to maintain consistency (self-attention)
  • Then they look at the original text to understand what to translate next (cross-attention)
  • Finally, they process all this information to produce the next part of the translation (feed-forward network)

For example, in machine translation, the decoder generates each word of the target language by:

  • Looking at the words it has already generated (to maintain grammatical coherence)
  • Looking at the encoded source sentence (to understand what content to translate)
  • Combining this information to produce the most appropriate next word

This architecture is powerful for tasks like translation, summarization, and text generation.

How It Works

A transformer decoder layer is a fundamental building block of transformer-based models for sequence-to-sequence tasks. It consists of three main components: a masked self-attention mechanism that processes the target sequence, a cross-attention mechanism that attends to the encoder’s output, and a feed-forward network for additional transformation. Each component is followed by layer normalization and residual connections to facilitate training of deep networks.

Constructors

Constructor Summary
TransformerDecoderLayer(int numHeads, int feedForwardDim, int sequenceLength, IActivationFunction<T> ffnActivation, double dropoutRate, bool causal) Lazy ctor: _embeddingSize is resolved from input.Shape[^1] on first Forward; the inner attention/cross-attention/ FFN/norm sublayers are constructed then.

Properties

Property Summary
IReadOnlyList<LayerPort> InputPorts { get; } Declares named input ports for this multi-input layer.
T AuxiliaryLossWeight { get; set; } Gets or sets the weight for the auxiliary loss contribution.
bool IsCausal { get; } Whether the self-attention is causal.
bool SupportsTraining { get; } Gets a value indicating whether this layer supports training.
bool UseAuxiliaryLoss { get; set; } Gets or sets a value indicating whether auxiliary loss is enabled for this layer.
double DropoutRate { get; } Dropout probability on each sublayer output.

Methods

Method Summary
Dictionary<string, string> GetAuxiliaryLossDiagnostics() Gets diagnostic information about the auxiliary loss computation.
Dictionary<string, string> GetDiagnostics() Gets diagnostic information about this component’s state and behavior.
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
T ComputeAuxiliaryLoss() Computes the auxiliary loss for this layer by aggregating losses from sublayers.
Tensor<T> Forward(Tensor<T> input, Tensor<T> encoderOutput) Performs the forward pass of the transformer decoder layer.
void ClearGradients()
void ResetState() Resets the internal state of the transformer decoder layer and all its sublayers.
void UpdateParameters(T learningRate) Updates the parameters of all sublayers using the calculated gradients.
void UpdateParametersGpu(IGpuOptimizerConfig config) Updates layer parameters using GPU-resident optimizer.