PreLNTransformerBlock<T>

Layers · AiDotNet.NeuralNetworks.Layers

Pre-Layer-Normalization transformer block with RMSNorm and a caller-supplied self-attention sublayer. Matches the decoder-style architecture used by T5, LLaMA, Gemma, Qwen2, and ChatGLM3 text encoders / language models.

Inheritance: LayerBase<T> → PreLNTransformerBlock<T>

Implements: IShapeContract

How It Works

Block structure (Raffel 2020 §2.1, Touvron 2023 §2.2):

The attention sublayer is injected through the constructor so the same block class supports T5’s relative-bias attention, Gemma’s RoPE multi-head attention, Qwen2’s RoPE grouped-query attention, and ChatGLM3’s RoPE multi-query attention. The FFN is a paper-canonical two-matrix linear → activation → linear stack with no biases (matching the T5 / LLaMA / Gemma / Qwen2 / ChatGLM3 convention).

Constructors

Constructor Summary
PreLNTransformerBlock(int hiddenSize, int ffnDim, LayerBase<T> attention, IActivationFunction<T> ffnActivation, bool gated) Initialises a pre-LN transformer block.

Properties

Property Summary
DenseLayer<T> FfnDown { get; } The FFN down-projection DenseLayer (ffnDim -> hidden, identity).
DenseLayer<T> FfnGate { get; } The FFN gate-projection DenseLayer (hidden -> ffnDim, activation), present only in gated SwiGLU mode; null for the classic two-matrix FFN.
DenseLayer<T> FfnUp { get; } The FFN up-projection DenseLayer (hidden -> ffnDim).
IActivationFunction<T> FfnActivation { get; } The FFN activation function applied after the up-projection.
LayerBase<T> AttentionLayer { get; } The self-attention sublayer (exposed for tensor-parallel serving partitioning).
RMSNormalizationLayer<T> Norm1 { get; } The pre-attention RMSNorm sublayer (exposed for tensor-parallel serving partitioning).
RMSNormalizationLayer<T> Norm2 { get; } The pre-FFN RMSNorm sublayer (exposed for tensor-parallel serving partitioning).
bool IsGated { get; } Whether the FFN is a gated SwiGLU (LLaMA/Mistral/Qwen2: down(act(gate(x)) * up(x))) rather than the classic two-matrix down(act(up(x))).
bool SupportsTraining { get; }
int FfnDim { get; } The FFN hidden dimension.
int HiddenSize { get; } The model (input/output) feature dimension.

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
void ClearGradients()
void ReplaceAttention(LayerBase<T> replacement) Swaps the attention sublayer in place (e.g.
void ResetState()
void UpdateParameters(T learningRate)