QwenDecoderLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

A Qwen-family causal decoder (Bai et al. 2023), the language model of GOT-OCR2: Qwen-0.5B. It reuses the repository’s LLaMA-style pre-RMSNorm blocks (PreLNTransformerBlock), with rotary grouped-query attention (biased q/k/v, as Qwen) and a SiLU-gated feed-forward. The head is tied to the token embedding.

Inheritance: LayerBase<T> → QwenDecoderLayer<T>

Implements: IShapeContract

How It Works

Image embeddings can be spliced into the token sequence at a given offset. GOT-OCR2’s prompt reserves <imgpad> slots for them, and the reference fills them with masked_scatter.

Constructors

Constructor Summary
QwenDecoderLayer(int vocabSize, int dim, int numHeads, int numKeyValueHeads, int ffnDim, int numLayers, int maxPositions, double ropeTheta, double rmsNormEpsilon)

Properties

Property Summary
IReadOnlyList<LayerPort> InputPorts { get; } Generated from [TensorPort] declarations.
IReadOnlyList<LayerPort> OutputPorts { get; } Generated from [TensorPort] declarations.
bool SupportsTraining { get; }
int Dim { get; }
int MaxPositions { get; } Longest sequence the rotary tables and attention cover.
int VocabSize { get; }

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
LayerInputDomain GetInputDomain(int[] inputShape) Generated from the primary [TensorPort] declaration.
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).