RotaryPositionalEncodingLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
Implements Rotary Position Embedding (RoPE) as described by Su et al., 2021.
Inheritance: LayerBase<T> → RotaryPositionalEncodingLayer<T>
Implements: IShapeContract
For Beginners
RoPE is the most popular position encoding for modern LLMs.
Instead of adding position information to embeddings (like sinusoidal encoding), RoPE rotates the query and key vectors based on their position. This means:
- The angle of rotation depends on the token’s position
- When two tokens attend to each other, the rotation encodes their relative distance
- This works naturally with KV-caching (just rotate new tokens at their position)
Used by Llama 2/3, Mistral, Phi-3, Gemma, and most modern LLMs.
How It Works
RoPE encodes absolute position by rotating query and key vectors in 2D subspaces using rotation matrices derived from position-dependent frequencies. The dot product of rotated queries and keys naturally captures relative position information.
The rotation formula for each pair of dimensions (2i, 2i+1):
Constructors
| Constructor | Summary |
|---|---|
RotaryPositionalEncodingLayer(int maxSequenceLength, int headDimension, double theta) |
Creates a new Rotary Position Embedding layer. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
double Theta { get; } |
Gets the base frequency parameter theta. |
int HeadDimension { get; } |
Gets the head dimension this RoPE layer operates on. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
ValueTuple<Tensor<T>, Tensor<T>> ApplyInverseRoPE(Tensor<T> gradQuery, Tensor<T> gradKey, int startPosition) |
Applies inverse RoPE rotation to query and key gradients (for backward pass). |
ValueTuple<Tensor<T>, Tensor<T>> ApplyRoPE(Tensor<T> queries, Tensor<T> keys, int startPosition) |
Applies RoPE rotation to query and key tensors. |
ValueTuple<Tensor<T>, Tensor<T>> GetInterleavedCaches(int requiredLength) |
Ensures the cache covers at least up to the specified sequence length. |
void ResetState() |