ALiBiPositionalBiasLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Implements Attention with Linear Biases (ALiBi) from Press et al., 2022.

Inheritance: LayerBase<T> → ALiBiPositionalBiasLayer<T>

Implements: IShapeContract

For Beginners

ALiBi is a simple way to encode position in attention.

Instead of adding position embeddings to tokens, ALiBi penalizes attention scores based on how far apart two tokens are:

  • Nearby tokens: small penalty (easy to attend to)
  • Far away tokens: large penalty (harder to attend to)
  • Different heads use different penalty strengths

Benefits:

  • No extra parameters to learn
  • Excellent length extrapolation (works well on longer sequences than training)
  • Very simple to implement and efficient

Used by BLOOM, MPT, and some Falcon variants.

How It Works

ALiBi adds a position-dependent linear bias to attention scores instead of modifying input embeddings. Each attention head uses a different slope, and the bias for head h between query position i and key position j is: bias[h, i, j] = -slope_h * |i - j|.

Per-head slopes follow a geometric sequence: slope_h = 2^(-8/numHeads * (h+1)). This ensures different heads attend at different distance scales, from very local to broader context.

Constructors

Constructor Summary
ALiBiPositionalBiasLayer(int numHeads, int maxSequenceLength) Creates a new ALiBi positional bias layer.

Properties

Property Summary
bool SupportsTraining { get; }
int NumHeads { get; } Gets the number of attention heads.

Methods

Method Summary
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
T[] GetSlopes() Gets the per-head slope values.
Tensor<T> ComputeBias(int queryLen, int keyLen, bool useCausalMask) Computes the ALiBi bias tensor for the given query and key lengths.
void ResetState()