CohereDecoderBlock<T>

Layers · AiDotNet.NeuralNetworks.Layers

Cohere (Command-R) decoder block with a parallel residual: a single LayerNorm feeds both the attention and the gated-SwiGLU FFN, whose outputs are added together to the residual — x = x + Attn(norm(x)) + FFN(norm(x)).

Inheritance: LayerBase<T> → CohereDecoderBlock<T>

Implements: IShapeContract

How It Works

Uses true LayerNorm (mean-centered) rather than RMSNorm, bias-free, matching Command-R. QK-normalization (present in some Command-R+ variants) is not applied.

Constructors

Constructor Summary
CohereDecoderBlock(int hiddenSize, int ffnDim, LayerBase<T> attention, double layerNormEpsilon) Creates a Cohere parallel-residual decoder block.

Properties

Property Summary
DenseLayer<T> FfnDown { get; } The FFN down projection.
DenseLayer<T> FfnGate { get; } The gated SwiGLU gate projection.
DenseLayer<T> FfnUp { get; } The gated SwiGLU up (value) projection.
LayerBase<T> AttentionLayer { get; } The self-attention sublayer.
LayerNormalizationLayer<T> Norm { get; } The single shared LayerNorm feeding both sublayers.
bool SupportsTraining { get; }
int HiddenSize { get; } The model (input/output) feature dimension.

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
void ClearGradients()
void ResetState()
void UpdateParameters(T learningRate)