ScaledPositionalEncodingLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Adds sinusoidal positional encodings scaled by a trainable weight: x_i + α · PE(i).

Inheritance: LayerBase<T> → ScaledPositionalEncodingLayer<T>

Implements: IShapeContract

For Beginners

Tells each position where it is in the sequence, with a learned volume knob for how loudly.

How It Works

Transformer TTS (Li et al. 2019, §3.2, Eq. 8): text embeddings and mel spectrograms live at different scales, so fixed positional embeddings “may impose heavy constraints on both the encoder and decoder pre-nets”; the triangle positional embeddings of Vaswani et al. (PE(pos, 2i) = sin(pos / 10000^(2i/d)), PE(pos, 2i+1) = cos(...)) are added with a trainable weight α, one for the encoder and one for the decoder. α starts at 1.

Input and output are [time, d] or [batch, time, d].

Constructors

Constructor Summary
ScaledPositionalEncodingLayer(int dimension) Creates the layer for features of width dimension.

Properties

Property Summary
T Alpha { get; } The current scale α.
bool SupportsTraining { get; }

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).