RelativePositionTransformerBlock<T>
Layers · AiDotNet.NeuralNetworks.Layers
A post-normalization Transformer encoder layer with relative position representations (Shaw et al. 2018) and a convolutional feed-forward network: Glow-TTS’s text encoder layer, shared by Grad-TTS, Matcha-TTS and VITS.
Inheritance: LayerBase<T> → RelativePositionTransformerBlock<T>
Implements: IShapeContract
For Beginners
Self-attention that knows how far apart two positions are rather than where each sits in the sentence, followed by a small convolution — a good fit for text, where nearby sounds matter most.
How It Works
Glow-TTS (Kim et al. 2020, §3.3, App. A.1): “we do not use absolute positional encoding but relative position
representations … the maximum relative position of 4”. Following the reference implementation
(attentions.Encoder, attentions.MultiHeadAttention, attentions.FFN):
With rotary set, positions enter through rotary embeddings instead (Su et al. 2021), applied to the first
half of each head’s query and key channels with the half-split pairing of Matcha-TTS’s encoder
(RotaryPositionalEmbeddings(k_channels * 0.5); Mehta et al. 2024), and no relative tables are used.
With positional cleared the attention carries no position information at all, as the reference
attentions.Encoder with window_size=None that VITS2 places in its normalizing flows (Kong et al. 2023,
§2.3).
Input and output are [time, hidden] or [batch, time, hidden].
Constructors
| Constructor | Summary |
|---|---|
RelativePositionTransformerBlock(int hidden, int heads, int filter, int kernelSize, double dropoutRate, int window, bool rotary, bool positional, double normalizationEpsilon) |
Creates the layer. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |