TransformerEncoderBlock<T>
Layers · AiDotNet.NeuralNetworks.Layers
Transformer encoder block — multi-head self-attention and a position-wise feed-forward network, each wrapped in a residual (skip) connection with layer normalization applied before the sublayer (Pre-LN, the default) or after the residual sum (Post-LN).
Inheritance: LayerBase<T> → TransformerEncoderBlock<T>
Implements: IShapeContract
For Beginners
A residual connection means “add the layer’s input back to its output.” It’s like keeping a copy of the original so nothing important gets lost as the data passes through. Transformers literally cannot learn without them.
How It Works
Block structure (Pre-LN, Xiong et al. 2020 “On Layer Normalization in the Transformer Architecture”; the residual/FFN design is Vaswani 2017 §3.1):
With PostNorm it is the original ordering instead, which
BERT and wav2vec 2.0 BASE use: y = LayerNorm(x + Dropout(SelfAttention(x))),
z = LayerNorm(y + Dropout(FFN(y))).
The residual connections (the x + / y + terms) are the
defining feature of the transformer: they let the input signal flow
unattenuated through arbitrarily deep stacks. Without them the attention/FFN
output REPLACES the hidden state each layer, the token-identity signal is
washed out (empirically ~60× per layer), and the network mode-collapses to
an input-independent constant output — the root cause of issue #1380.
Pre-LN vs Post-LN: normalizing the sublayer INPUT (Pre-LN) rather than the residual SUM (Post-LN, the original 2017 ordering) keeps the residual path un-normalized, so gradients flow cleanly through depth and the model trains stably WITHOUT learning-rate warmup. Post-LN converges far slower without warmup; Pre-LN is the ordering used by every modern transformer (GPT-2 onward, LLaMA, etc.) and trains measurably faster here.
Constructors
| Constructor | Summary |
|---|---|
TransformerEncoderBlock(int hiddenSize, int numHeads, int ffnDim, double dropoutRate, IActivationFunction<T> ffnActivation, TransformerNormPlacement normPlacement) |
Initialises a transformer encoder block. |
Properties
| Property | Summary |
|---|---|
LayerBase<T> AttentionLayer { get; } |
The block’s current self-attention sublayer. |
LayerBase<T> FfnDownLayer { get; } |
The block’s current FFN down-projection (ffnDim → hiddenSize) sublayer. |
LayerBase<T> FfnUpLayer { get; } |
The block’s current FFN up-projection (hiddenSize → ffnDim) sublayer. |
TransformerNormPlacement NormPlacement { get; } |
Where each LayerNorm sits relative to its residual connection — persisted for deserialization. |
bool SupportsTraining { get; } |
|
double DropoutRate { get; } |
Dropout probability — persisted for deserialization. |
int FfnDim { get; } |
Feed-forward inner dimension — persisted for deserialization. |
int HiddenSize { get; } |
Model (feature) dimension — persisted for deserialization. |
int NumHeads { get; } |
Number of attention heads — persisted for deserialization. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
void ClearGradients() |
|
void ReplaceAttention(LayerBase<T> replacement) |
Swaps the block’s self-attention sublayer for replacement — the composite-layer counterpart of the inference optimizer assigning a rewritten attention layer into model.Layers[i] for discrete layouts. |
void ReplaceFfnDown(LayerBase<T> replacement) |
Swaps the FFN down-projection sublayer (e.g. |
void ReplaceFfnUp(LayerBase<T> replacement) |
Swaps the FFN up-projection sublayer (e.g. |
void ResetState() |
|
void SetTrainingMode(bool isTraining) |
|
void UpdateParameters(T learningRate) |