BASEDLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers.SSM
Implements the BASED (Bidirectional Attention with Sliding-window and Expanded features) layer from “Simple Linear Attention Language Models Balance the Recall-Throughput Tradeoff” (Arora et al., 2024).
Inheritance: LayerBase<T> → BASEDLayer<T>
Implements: IShapeContract
For Beginners
BASED combines two ways of paying attention to solve a key tradeoff:
Imagine you’re reading a long book and need to answer questions about it:
- Linear attention is like having a summary of everything you’ve read so far. It’s fast to update
and gives you the gist, but you might miss exact details (like a specific name on page 42).
- Sliding window attention is like being able to flip back a few pages to check exact wording.
It’s precise but only covers recent pages.
BASED combines both: the summary (linear attention) gives global context, and the “flip back” (sliding window) handles precise recall of recent information. Together, they match the quality of full Transformer attention while being much more efficient for long sequences.
The Taylor expansion feature map phi(x) = [1, x, xx/sqrt(2)] is a mathematical trick that approximates the softmax kernel exp(qk) using polynomial terms. The constant 1 captures baseline attention, x captures first-order similarity, and x*x/sqrt(2) captures quadratic interactions.
How It Works
BASED is a hybrid architecture that combines two attention mechanisms to get the best of both worlds:
The key insight is that linear attention is fast (O(n)) but struggles with recall-intensive tasks because the feature map approximation loses precision. Adding a small sliding window (w=64 or 128) cheaply fixes this: the window handles precise local recall while linear attention covers global context. The total complexity is O(n * (d + w)) which is still linear in sequence length for fixed window size.
Reference: Arora et al., “Simple Linear Attention Language Models Balance the Recall-Throughput Tradeoff”, 2024.
https://arxiv.org/abs/2402.18668
Constructors
| Constructor | Summary |
|---|---|
BASEDLayer(int sequenceLength, int modelDimension, int numHeads, int windowSize, int featureExpansion, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) |
Creates a new BASED layer that combines linear attention with sliding window attention. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int FeatureExpansion { get; } |
Gets the feature expansion factor for the Taylor feature map. |
int HeadDimension { get; } |
Gets the dimension per head. |
int ModelDimension { get; } |
Gets the model dimension. |
int NumHeads { get; } |
Gets the number of attention heads. |
int WindowSize { get; } |
Gets the sliding window size for local attention. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
Tensor<T> GetFeatureMapScale() |
Gets the feature map scale parameters for external inspection. |
Tensor<T> GetLinearQueryWeights() |
Gets the linear attention query weights for external inspection. |
Tensor<T> GetOutputProjectionWeights() |
Gets the output projection weights for external inspection. |
Tensor<T> GetWindowQueryWeights() |
Gets the window attention query weights for external inspection. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |