T5RelativeBiasAttentionLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
T5-style multi-head self-attention with learned relative position bias (Raffel et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”, JMLR 2020).
Inheritance: LayerBase<T> → T5RelativeBiasAttentionLayer<T>
Implements: IShapeContract
For Beginners
Standard attention layers learn position information only through fixed sinusoidal patterns added to the input. T5 instead learns directly how much to bias attention between any two positions — a richer, fully-trained position signal that has been a major contributor to T5’s strong empirical performance.
How It Works
Two paper-faithful deviations from the standard MultiHeadAttention layer:
Shared-bias convention: The original T5 paper shares ONE relative-
position bias table across all encoder layers (Raffel 2020 §2.1 footnote 5,
“the relative position embedding is shared across all layers but each head
has its own embedding”). The constructor accepts an optional
sharedRelativeBiasTable; when supplied, this layer reuses it instead
of allocating its own. The LayerHelper.CreateDefaultT5TextLayers
factory wires one shared table through every T5 attention layer in the
stack — that is the paper-canonical configuration. Standalone construction
(e.g. unit tests) gives each layer its own bias table, which is the
“common-but-non-canonical” HuggingFace T5 default.
Constructors
| Constructor | Summary |
|---|---|
T5RelativeBiasAttentionLayer(int hiddenSize, int numHeads, int numBuckets, int maxDistance, bool bidirectional, int? seed, Tensor<T> sharedRelativeBiasTable, bool usesExternalPositionBias, int? keyValueDim) |
Initialises a new T5-style relative-bias self-attention layer. |
Properties
| Property | Summary |
|---|---|
bool OwnsRelativeBiasTable { get; } |
Returns whether this layer owns its bias table (vs. |
bool SupportsTraining { get; } |
|
bool UsesExternalPositionBias { get; } |
Whether this layer receives its relative position bias from its stack. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
Tensor<T> GetRelativeBiasTable() |
Gets the relative bias table for testing and inspection. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |