RelativePositionTransformerBlock<T>

Layers · AiDotNet.NeuralNetworks.Layers

A post-normalization Transformer encoder layer with relative position representations (Shaw et al. 2018) and a convolutional feed-forward network: Glow-TTS’s text encoder layer, shared by Grad-TTS, Matcha-TTS and VITS.

Inheritance: LayerBase<T> → RelativePositionTransformerBlock<T>

Implements: IShapeContract

For Beginners

Self-attention that knows how far apart two positions are rather than where each sits in the sentence, followed by a small convolution — a good fit for text, where nearby sounds matter most.

How It Works

Glow-TTS (Kim et al. 2020, §3.3, App. A.1): “we do not use absolute positional encoding but relative position representations … the maximum relative position of 4”. Following the reference implementation (attentions.Encoder, attentions.MultiHeadAttention, attentions.FFN):

With rotary set, positions enter through rotary embeddings instead (Su et al. 2021), applied to the first half of each head’s query and key channels with the half-split pairing of Matcha-TTS’s encoder (RotaryPositionalEmbeddings(k_channels * 0.5); Mehta et al. 2024), and no relative tables are used.

With positional cleared the attention carries no position information at all, as the reference attentions.Encoder with window_size=None that VITS2 places in its normalizing flows (Kong et al. 2023, §2.3).

Input and output are [time, hidden] or [batch, time, hidden].

Constructors

Constructor Summary
RelativePositionTransformerBlock(int hidden, int heads, int filter, int kernelSize, double dropoutRate, int window, bool rotary, bool positional, double normalizationEpsilon) Creates the layer.

Properties

Property Summary
bool SupportsTraining { get; }

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).