TransNormerLLMLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers.SSM
Implements the TransNormerLLM layer from “TransNormerLLM: A Faster and Better LLM” (Qin et al., 2023).
Inheritance: LayerBase<T> → TransNormerLLMLayer<T>
Implements: IShapeContract
For Beginners
TransNormerLLM makes “linear attention” actually work well for language models.
Standard linear attention has a known problem: it tends to become numerically unstable during training because the accumulated state matrix can grow without bound. TransNormerLLM fixes this with two tricks:
- RMSNorm on Q and K: Before computing attention, the queries and keys are normalized. This is like
making sure all “questions” and “answers” have similar magnitude, preventing any single token from dominating the accumulated state.
- Exponential decay: Old information naturally fades away (controlled by gamma), preventing the state
from accumulating indefinitely. This is similar to RetNet’s approach but simpler.
Together, these allow TransNormerLLM to match or exceed Transformer quality while being much faster for long sequences (linear vs quadratic complexity).
How It Works
TransNormerLLM uses “Lightning Attention” – a linear attention variant with exponential decay and efficient normalization. Unlike standard Transformers that use softmax attention (O(n^2)), Lightning Attention achieves linear complexity O(n) by combining linear attention with a decay factor and RMSNorm-based normalization.
The architecture:
The key innovations over standard linear attention:
- RMSNorm on Q and K prevents the magnitude explosion that plagues linear attention
- Exponential decay gamma provides a soft causal bias (like RetNet) without rotary PE
- Per-output RMSNorm stabilizes the attention output, preventing training instability
These simple modifications make linear attention competitive with softmax attention for LLMs.
Reference: Qin et al., “TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer”, 2023.
https://arxiv.org/abs/2307.14995
Constructors
| Constructor | Summary |
|---|---|
TransNormerLLMLayer(int sequenceLength, int modelDimension, int numHeads, double decayRate, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) |
Creates a new TransNormerLLM layer with lightning attention. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
double DecayRate { get; } |
Gets the decay rate. |
int HeadDimension { get; } |
Gets the dimension per head. |
int ModelDimension { get; } |
Gets the model dimension. |
int NumHeads { get; } |
Gets the number of attention heads. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
Tensor<T> GetDecayRates() |
Gets the per-head decay rates (gammas) for external inspection. |
Tensor<T> GetOutputProjectionWeights() |
Gets the output projection weights for external inspection. |
Tensor<T> GetQueryWeights() |
Gets the query weights for external inspection. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |