PaTHAttentionLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers.SSM
Implements the PaTH Attention (Positional-aware Transformer via Householder) layer from Mao et al., 2025 (arXiv:2505.16381).
Inheritance: LayerBase<T> → PaTHAttentionLayer<T>
Implements: IShapeContract
For Beginners
PaTH Attention is a new way to tell the model WHERE each token is in the sequence, using geometry instead of adding numbers.
Traditional approaches add position information:
- “I am token 5” gets added as a number pattern to the token’s embedding
- This can interfere with the token’s meaning
PaTH instead REFLECTS the token’s query/key vectors using a mirror unique to each position:
- Position 1 has mirror A, position 2 has mirror B, etc.
- Each mirror “rotates” the query/key differently based on position
- The attention score naturally captures both WHAT the token means AND WHERE it is
Think of it like a hall of mirrors:
- Each position has its own unique mirror (Householder reflection)
- When you look at a token through its position’s mirror, you see a unique view
- Two tokens at different positions produce different reflections
- The similarity between reflections captures both content and position
This is mathematically cleaner because reflections preserve vector lengths (they just change direction), whereas adding position numbers changes lengths and can distort meaning.
How It Works
PaTH Attention replaces traditional positional encodings (sinusoidal, rotary, etc.) with Householder reflections applied to queries and keys. Each position in the sequence has a learned reflection vector p, and the corresponding Householder transform H = I - 2pp^T/||p||^2 is applied to Q and K before computing attention. This embeds positional information directly into the geometry of the attention space rather than adding it as a bias.
The architecture:
The key insight is that Householder reflections are orthogonal transformations. Unlike additive positional encodings that can distort the magnitude of embeddings, Householder reflections preserve norms while encoding position. The attention score between two positions depends on BOTH the content (Q, K) and the relative position (H_i vs H_j), but in a multiplicative way that is more expressive than simple additive bias.
Reference: Mao et al., “PaTH Attention: Positional-aware Transformer via Householder”, 2025.
https://arxiv.org/abs/2505.16381
Constructors
| Constructor | Summary |
|---|---|
PaTHAttentionLayer(int sequenceLength, int modelDimension, int numHeads, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) |
Creates a new PaTH Attention layer. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int HeadDimension { get; } |
Gets the dimension per head. |
int ModelDimension { get; } |
Gets the model dimension. |
int NumHeads { get; } |
Gets the number of attention heads. |
int SequenceLength { get; } |
Gets the sequence length. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns the complete convention-enumerated trainable tensor surface. |
Tensor<T> GetHouseholderVectors() |
Gets the Householder vectors for external inspection. |
Tensor<T> GetOutputProjectionWeights() |
Gets the output projection weights for external inspection. |
Tensor<T> GetQueryWeights() |
Gets the query weights for external inspection. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
|
void UpdateParameters(T learningRate) |