TransformerEncoderBlock<T>

Layers · AiDotNet.NeuralNetworks.Layers

Transformer encoder block — multi-head self-attention and a position-wise feed-forward network, each wrapped in a residual (skip) connection with layer normalization applied before the sublayer (Pre-LN, the default) or after the residual sum (Post-LN).

Inheritance: LayerBase<T> → TransformerEncoderBlock<T>

Implements: IShapeContract

For Beginners

A residual connection means “add the layer’s input back to its output.” It’s like keeping a copy of the original so nothing important gets lost as the data passes through. Transformers literally cannot learn without them.

How It Works

Block structure (Pre-LN, Xiong et al. 2020 “On Layer Normalization in the Transformer Architecture”; the residual/FFN design is Vaswani 2017 §3.1):

With PostNorm it is the original ordering instead, which BERT and wav2vec 2.0 BASE use: y = LayerNorm(x + Dropout(SelfAttention(x))), z = LayerNorm(y + Dropout(FFN(y))).

The residual connections (the x + / y + terms) are the defining feature of the transformer: they let the input signal flow unattenuated through arbitrarily deep stacks. Without them the attention/FFN output REPLACES the hidden state each layer, the token-identity signal is washed out (empirically ~60× per layer), and the network mode-collapses to an input-independent constant output — the root cause of issue #1380.

Pre-LN vs Post-LN: normalizing the sublayer INPUT (Pre-LN) rather than the residual SUM (Post-LN, the original 2017 ordering) keeps the residual path un-normalized, so gradients flow cleanly through depth and the model trains stably WITHOUT learning-rate warmup. Post-LN converges far slower without warmup; Pre-LN is the ordering used by every modern transformer (GPT-2 onward, LLaMA, etc.) and trains measurably faster here.

Constructors

Constructor Summary
TransformerEncoderBlock(int hiddenSize, int numHeads, int ffnDim, double dropoutRate, IActivationFunction<T> ffnActivation, TransformerNormPlacement normPlacement) Initialises a transformer encoder block.

Properties

Property Summary
LayerBase<T> AttentionLayer { get; } The block’s current self-attention sublayer.
LayerBase<T> FfnDownLayer { get; } The block’s current FFN down-projection (ffnDim → hiddenSize) sublayer.
LayerBase<T> FfnUpLayer { get; } The block’s current FFN up-projection (hiddenSize → ffnDim) sublayer.
TransformerNormPlacement NormPlacement { get; } Where each LayerNorm sits relative to its residual connection — persisted for deserialization.
bool SupportsTraining { get; }
double DropoutRate { get; } Dropout probability — persisted for deserialization.
int FfnDim { get; } Feed-forward inner dimension — persisted for deserialization.
int HiddenSize { get; } Model (feature) dimension — persisted for deserialization.
int NumHeads { get; } Number of attention heads — persisted for deserialization.

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
void ClearGradients()
void ReplaceAttention(LayerBase<T> replacement) Swaps the block’s self-attention sublayer for replacement — the composite-layer counterpart of the inference optimizer assigning a rewritten attention layer into model.Layers[i] for discrete layouts.
void ReplaceFfnDown(LayerBase<T> replacement) Swaps the FFN down-projection sublayer (e.g.
void ReplaceFfnUp(LayerBase<T> replacement) Swaps the FFN up-projection sublayer (e.g.
void ResetState()
void SetTrainingMode(bool isTraining)
void UpdateParameters(T learningRate)