Gemma2DecoderBlock<T>
Layers · AiDotNet.NeuralNetworks.Layers
Gemma-2 decoder block with sandwiched RMSNorms — a norm both before and after each sublayer:
x = x + postAttnNorm(Attn(inputNorm(x))) then x = x + postFfnNorm(GeGLU(preFfnNorm(x))).
The FFN is a gated GeGLU (tanh-GELU gate); RMSNorms use the Gemma (1 + weight) convention (applied
at load time).
Inheritance: LayerBase<T> → Gemma2DecoderBlock<T>
Implements: IShapeContract
Constructors
| Constructor | Summary |
|---|---|
Gemma2DecoderBlock(int hiddenSize, int ffnDim, LayerBase<T> attention, double rmsNormEpsilon) |
Creates a Gemma-2 decoder block. |
Properties
| Property | Summary |
|---|---|
DenseLayer<T> FfnDown { get; } |
The FFN down projection. |
DenseLayer<T> FfnGate { get; } |
The GeGLU gate projection. |
DenseLayer<T> FfnUp { get; } |
The GeGLU up (value) projection. |
LayerBase<T> AttentionLayer { get; } |
The self-attention sublayer. |
RMSNormalizationLayer<T> NormInput { get; } |
Pre-attention RMSNorm. |
RMSNormalizationLayer<T> NormPostAttn { get; } |
Post-attention RMSNorm (normalizes the attention output before the residual add). |
RMSNormalizationLayer<T> NormPostFfn { get; } |
Post-FFN RMSNorm (normalizes the FFN output before the residual add). |
RMSNormalizationLayer<T> NormPreFfn { get; } |
Pre-FFN RMSNorm. |
bool SupportsTraining { get; } |
|
int HiddenSize { get; } |
The model (input/output) feature dimension. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
void ClearGradients() |
|
void ResetState() |
|
void UpdateParameters(T learningRate) |