ALiBiPositionalBiasLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
Implements Attention with Linear Biases (ALiBi) from Press et al., 2022.
Inheritance: LayerBase<T> → ALiBiPositionalBiasLayer<T>
Implements: IShapeContract
For Beginners
ALiBi is a simple way to encode position in attention.
Instead of adding position embeddings to tokens, ALiBi penalizes attention scores based on how far apart two tokens are:
- Nearby tokens: small penalty (easy to attend to)
- Far away tokens: large penalty (harder to attend to)
- Different heads use different penalty strengths
Benefits:
- No extra parameters to learn
- Excellent length extrapolation (works well on longer sequences than training)
- Very simple to implement and efficient
Used by BLOOM, MPT, and some Falcon variants.
How It Works
ALiBi adds a position-dependent linear bias to attention scores instead of modifying input embeddings. Each attention head uses a different slope, and the bias for head h between query position i and key position j is: bias[h, i, j] = -slope_h * |i - j|.
Per-head slopes follow a geometric sequence: slope_h = 2^(-8/numHeads * (h+1)). This ensures different heads attend at different distance scales, from very local to broader context.
Constructors
| Constructor | Summary |
|---|---|
ALiBiPositionalBiasLayer(int numHeads, int maxSequenceLength) |
Creates a new ALiBi positional bias layer. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int NumHeads { get; } |
Gets the number of attention heads. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
T[] GetSlopes() |
Gets the per-head slope values. |
Tensor<T> ComputeBias(int queryLen, int keyLen, bool useCausalMask) |
Computes the ALiBi bias tensor for the given query and key lengths. |
void ResetState() |