Gemma2DecoderBlock<T>

Layers · AiDotNet.NeuralNetworks.Layers

Gemma-2 decoder block with sandwiched RMSNorms — a norm both before and after each sublayer: x = x + postAttnNorm(Attn(inputNorm(x))) then x = x + postFfnNorm(GeGLU(preFfnNorm(x))). The FFN is a gated GeGLU (tanh-GELU gate); RMSNorms use the Gemma (1 + weight) convention (applied at load time).

Inheritance: LayerBase<T> → Gemma2DecoderBlock<T>

Implements: IShapeContract

Constructors

Constructor Summary
Gemma2DecoderBlock(int hiddenSize, int ffnDim, LayerBase<T> attention, double rmsNormEpsilon) Creates a Gemma-2 decoder block.

Properties

Property Summary
DenseLayer<T> FfnDown { get; } The FFN down projection.
DenseLayer<T> FfnGate { get; } The GeGLU gate projection.
DenseLayer<T> FfnUp { get; } The GeGLU up (value) projection.
LayerBase<T> AttentionLayer { get; } The self-attention sublayer.
RMSNormalizationLayer<T> NormInput { get; } Pre-attention RMSNorm.
RMSNormalizationLayer<T> NormPostAttn { get; } Post-attention RMSNorm (normalizes the attention output before the residual add).
RMSNormalizationLayer<T> NormPostFfn { get; } Post-FFN RMSNorm (normalizes the FFN output before the residual add).
RMSNormalizationLayer<T> NormPreFfn { get; } Pre-FFN RMSNorm.
bool SupportsTraining { get; }
int HiddenSize { get; } The model (input/output) feature dimension.

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
void ClearGradients()
void ResetState()
void UpdateParameters(T learningRate)