RMSNormalizationLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
Root-Mean-Square Layer Normalization (Zhang & Sennrich 2019).
Inheritance: LayerBase<T> → RMSNormalizationLayer<T>
Implements: IShapeContract
For Beginners
RMSNorm is a simpler, faster cousin of LayerNorm.
LayerNorm: subtracts the mean, divides by standard deviation, then scales and shifts. RMSNorm: skips the mean-subtraction step, divides by the root-mean-square, then scales.
The “skip the mean” part isn’t a shortcut — Zhang & Sennrich showed it works just as well in practice but costs less compute. Every modern LLM-style text encoder (T5, LLaMA, Gemma, Qwen, ChatGLM) uses RMSNorm because it’s both paper-validated and faster.
How It Works
RMSNorm rescales each sample’s feature vector by its root-mean-square magnitude (without re-centering on the mean), then multiplies by a learnable per-feature gain γ. Concretely:
RMSNorm(x)_i = (x_i / sqrt(mean(x²) + ε)) · γ_i
This is the normalization used by T5 (Raffel et al. 2020), LLaMA (Touvron et al. 2023), Gemma (Gemma Team 2024), Qwen2 (Yang et al. 2024), and ChatGLM3 (Zeng et al. 2023). Unlike standard LayerNorm there is NO learnable shift β — the paper-canonical RMSNorm formulation only includes a multiplicative gain.
Reference: Zhang & Sennrich, “Root Mean Square Layer Normalization”, NeurIPS 2019.
Constructors
| Constructor | Summary |
|---|---|
RMSNormalizationLayer(double epsilon) |
Initializes a new instance of the RMSNormalizationLayer class. |
RMSNormalizationLayer(int featureSize, double epsilon) |
AiDotNet#1370 eager-init constructor. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
T GetEpsilon() |
Gets the epsilon value used for numerical stability. |
Tensor<T> GetGammaTensor() |
Gets the gain (γ) tensor for JIT compilation and internal use. |
Vector<T> GetGamma() |
Gets the gain (γ) parameters of the layer. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |