CohereDecoderBlock<T>
Layers · AiDotNet.NeuralNetworks.Layers
Cohere (Command-R) decoder block with a parallel residual: a single LayerNorm feeds both the
attention and the gated-SwiGLU FFN, whose outputs are added together to the residual —
x = x + Attn(norm(x)) + FFN(norm(x)).
Inheritance: LayerBase<T> → CohereDecoderBlock<T>
Implements: IShapeContract
How It Works
Uses true LayerNorm (mean-centered) rather than RMSNorm, bias-free, matching Command-R. QK-normalization (present in some Command-R+ variants) is not applied.
Constructors
| Constructor | Summary |
|---|---|
CohereDecoderBlock(int hiddenSize, int ffnDim, LayerBase<T> attention, double layerNormEpsilon) |
Creates a Cohere parallel-residual decoder block. |
Properties
| Property | Summary |
|---|---|
DenseLayer<T> FfnDown { get; } |
The FFN down projection. |
DenseLayer<T> FfnGate { get; } |
The gated SwiGLU gate projection. |
DenseLayer<T> FfnUp { get; } |
The gated SwiGLU up (value) projection. |
LayerBase<T> AttentionLayer { get; } |
The self-attention sublayer. |
LayerNormalizationLayer<T> Norm { get; } |
The single shared LayerNorm feeding both sublayers. |
bool SupportsTraining { get; } |
|
int HiddenSize { get; } |
The model (input/output) feature dimension. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
void ClearGradients() |
|
void ResetState() |
|
void UpdateParameters(T learningRate) |