StarCoder2DecoderBlock<T>
Layers · AiDotNet.NeuralNetworks.Layers
StarCoder2 pre-LN decoder block: y = x + Attn(LayerNorm(x)) then z = y + MLP(LayerNorm(y)),
where the norms are true (mean-centered) LayerNorms with bias, the attention projections carry
biases, and the FFN is a non-gated two-matrix GELU MLP (c_proj(gelu(c_fc(x)))) with biases.
Inheritance: LayerBase<T> → StarCoder2DecoderBlock<T>
Implements: IShapeContract
Constructors
| Constructor | Summary |
|---|---|
StarCoder2DecoderBlock(int hiddenSize, int ffnDim, LayerBase<T> attention, double layerNormEpsilon) |
Creates a StarCoder2 decoder block. |
Properties
| Property | Summary |
|---|---|
DenseLayer<T> CFc { get; } |
The FFN up projection (c_fc: hidden -> ffnDim, GELU, with bias). |
DenseLayer<T> CProj { get; } |
The FFN down projection (c_proj: ffnDim -> hidden, with bias). |
LayerBase<T> AttentionLayer { get; } |
The self-attention sublayer (with projection biases). |
LayerNormalizationLayer<T> Norm1 { get; } |
Pre-attention LayerNorm (with bias). |
LayerNormalizationLayer<T> Norm2 { get; } |
Pre-MLP LayerNorm (with bias). |
bool SupportsTraining { get; } |
|
int HiddenSize { get; } |
The model (input/output) feature dimension. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
void ClearGradients() |
|
void ResetState() |
|
void UpdateParameters(T learningRate) |