StarCoder2DecoderBlock<T>

Layers · AiDotNet.NeuralNetworks.Layers

StarCoder2 pre-LN decoder block: y = x + Attn(LayerNorm(x)) then z = y + MLP(LayerNorm(y)), where the norms are true (mean-centered) LayerNorms with bias, the attention projections carry biases, and the FFN is a non-gated two-matrix GELU MLP (c_proj(gelu(c_fc(x)))) with biases.

Inheritance: LayerBase<T> → StarCoder2DecoderBlock<T>

Implements: IShapeContract

Constructors

Constructor Summary
StarCoder2DecoderBlock(int hiddenSize, int ffnDim, LayerBase<T> attention, double layerNormEpsilon) Creates a StarCoder2 decoder block.

Properties

Property Summary
DenseLayer<T> CFc { get; } The FFN up projection (c_fc: hidden -> ffnDim, GELU, with bias).
DenseLayer<T> CProj { get; } The FFN down projection (c_proj: ffnDim -> hidden, with bias).
LayerBase<T> AttentionLayer { get; } The self-attention sublayer (with projection biases).
LayerNormalizationLayer<T> Norm1 { get; } Pre-attention LayerNorm (with bias).
LayerNormalizationLayer<T> Norm2 { get; } Pre-MLP LayerNorm (with bias).
bool SupportsTraining { get; }
int HiddenSize { get; } The model (input/output) feature dimension.

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
void ClearGradients()
void ResetState()
void UpdateParameters(T learningRate)