QwenDecoderLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
A Qwen-family causal decoder (Bai et al. 2023), the language model of GOT-OCR2: Qwen-0.5B. It reuses the
repository’s LLaMA-style pre-RMSNorm blocks (PreLNTransformerBlock), with rotary
grouped-query attention (biased q/k/v, as Qwen) and a SiLU-gated feed-forward. The head is tied to the
token embedding.
Inheritance: LayerBase<T> → QwenDecoderLayer<T>
Implements: IShapeContract
How It Works
Image embeddings can be spliced into the token sequence at a given offset. GOT-OCR2’s prompt reserves
<imgpad> slots for them, and the reference fills them with masked_scatter.
Constructors
| Constructor | Summary |
|---|---|
QwenDecoderLayer(int vocabSize, int dim, int numHeads, int numKeyValueHeads, int ffnDim, int numLayers, int maxPositions, double ropeTheta, double rmsNormEpsilon) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<LayerPort> InputPorts { get; } |
Generated from [TensorPort] declarations. |
IReadOnlyList<LayerPort> OutputPorts { get; } |
Generated from [TensorPort] declarations. |
bool SupportsTraining { get; } |
|
int Dim { get; } |
|
int MaxPositions { get; } |
Longest sequence the rotary tables and attention cover. |
int VocabSize { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
|
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
LayerInputDomain GetInputDomain(int[] inputShape) |
Generated from the primary [TensorPort] declaration. |
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |