LogLinearAttentionLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers.SSM

Implements the Log-Linear Attention layer from Zhang et al., 2025.

Inheritance: LayerBase<T> → LogLinearAttentionLayer<T>

Implements: IShapeContract

For Beginners

Think of this like a hierarchical note-taking system:

  • Level 0 (seconds): Detailed notes from the last few moments
  • Level 1 (minutes): Summaries of recent detailed notes
  • Level 2 (hours): High-level summaries of minute-level notes
  • Level 3 (days): Very compressed overviews

When you need to answer a question (query), you check all levels: recent details for recent questions, older summaries for historical questions.

The total storage is O(L * d^2) where L = log(n) levels, instead of O(n * d^2) for standard linear attention. This makes it much more memory-efficient for long sequences while retaining more information than a single fixed-size state.

How It Works

Standard linear attention maintains a hidden state that grows linearly with sequence length: at each step, a new key-value outer product is added to the state matrix S. Over long sequences, the accumulated state becomes noisy and the effective capacity is diluted, which is why linear attention often underperforms softmax attention on long contexts.

Log-Linear Attention solves this by organizing the state into a hierarchical multi-level structure where the total state size grows only logarithmically with sequence length. The key idea is periodic compression: instead of keeping every update, lower-level states are periodically compressed and promoted to higher levels.

The hierarchy works as follows:

Compression at each level uses a learned linear projection to reduce the accumulated state, preserving the most important information while discarding noise. This is analogous to how human memory works: recent events are stored in detail (Level 0), while older events are remembered as compressed summaries (higher levels).

Reference: Zhang et al., “Log-Linear Attention”, 2025. https://arxiv.org/abs/2506.04761

Constructors

Constructor Summary
LogLinearAttentionLayer(int sequenceLength, int modelDimension, int numHeads, int numLevels, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) Creates a new Log-Linear Attention layer with hierarchical state compression.

Properties

Property Summary
bool SupportsTraining { get; }
int HeadDimension { get; } Gets the dimension per head.
int ModelDimension { get; } Gets the model dimension.
int NumHeads { get; } Gets the number of heads.
int NumLevels { get; } Gets the number of hierarchy levels.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Tensor<T> GetLevelMixWeights() Gets the level mixing weights for external inspection.
Tensor<T> GetOutputProjectionWeights() Gets the output projection weights for external inspection.
Tensor<T> GetQueryWeights() Gets the query weights for external inspection.
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).