PreLNTransformerBlock<T>
Layers · AiDotNet.NeuralNetworks.Layers
Pre-Layer-Normalization transformer block with RMSNorm and a caller-supplied self-attention sublayer. Matches the decoder-style architecture used by T5, LLaMA, Gemma, Qwen2, and ChatGLM3 text encoders / language models.
Inheritance: LayerBase<T> → PreLNTransformerBlock<T>
Implements: IShapeContract
How It Works
Block structure (Raffel 2020 §2.1, Touvron 2023 §2.2):
The attention sublayer is injected through the constructor so the same block class supports T5’s relative-bias attention, Gemma’s RoPE multi-head attention, Qwen2’s RoPE grouped-query attention, and ChatGLM3’s RoPE multi-query attention. The FFN is a paper-canonical two-matrix linear → activation → linear stack with no biases (matching the T5 / LLaMA / Gemma / Qwen2 / ChatGLM3 convention).
Constructors
| Constructor | Summary |
|---|---|
PreLNTransformerBlock(int hiddenSize, int ffnDim, LayerBase<T> attention, IActivationFunction<T> ffnActivation, bool gated) |
Initialises a pre-LN transformer block. |
Properties
| Property | Summary |
|---|---|
DenseLayer<T> FfnDown { get; } |
The FFN down-projection DenseLayer (ffnDim -> hidden, identity). |
DenseLayer<T> FfnGate { get; } |
The FFN gate-projection DenseLayer (hidden -> ffnDim, activation), present only in gated SwiGLU mode; null for the classic two-matrix FFN. |
DenseLayer<T> FfnUp { get; } |
The FFN up-projection DenseLayer (hidden -> ffnDim). |
IActivationFunction<T> FfnActivation { get; } |
The FFN activation function applied after the up-projection. |
LayerBase<T> AttentionLayer { get; } |
The self-attention sublayer (exposed for tensor-parallel serving partitioning). |
RMSNormalizationLayer<T> Norm1 { get; } |
The pre-attention RMSNorm sublayer (exposed for tensor-parallel serving partitioning). |
RMSNormalizationLayer<T> Norm2 { get; } |
The pre-FFN RMSNorm sublayer (exposed for tensor-parallel serving partitioning). |
bool IsGated { get; } |
Whether the FFN is a gated SwiGLU (LLaMA/Mistral/Qwen2: down(act(gate(x)) * up(x))) rather than the classic two-matrix down(act(up(x))). |
bool SupportsTraining { get; } |
|
int FfnDim { get; } |
The FFN hidden dimension. |
int HiddenSize { get; } |
The model (input/output) feature dimension. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
void ClearGradients() |
|
void ReplaceAttention(LayerBase<T> replacement) |
Swaps the attention sublayer in place (e.g. |
void ResetState() |
|
void UpdateParameters(T learningRate) |