ABCLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers.SSM

Implements the ABC (Attention with Bounded-memory Control) layer from Peng et al., 2022.

Inheritance: LayerBase<T> → ABCLayer<T>

Implements: IShapeContract

For Beginners

ABC is like having a fixed number of filing cabinet drawers (slots) for storing information as you read through a long document.

Imagine you have 32 drawers and you’re reading a book:

  • At each word, you decide which drawers are most relevant (via attention scores)
  • You file information about the word into those drawers (competitive writing)
  • Old information gradually fades from drawers (forget gate)
  • When you need to answer a question, you look through the drawers (reading)

The “competitive” part is crucial: if many words want to use the same drawer, softmax ensures the most relevant one gets priority. This is what “bounded-memory control” means – you never need more drawers than the fixed number, no matter how long the book is.

Compare this to:

  • Standard attention: You keep all words accessible (expensive for long books)
  • Linear attention: You maintain a summary matrix (unbounded growth in rank)
  • ABC: You maintain exactly numSlots drawers of information (bounded)

How It Works

ABC uses a fixed-size set of memory “slots” with a competitive attention mechanism. Input tokens compete for writing to slots via softmax attention scores, and a forget mechanism clears stale slot content. This bounds memory usage regardless of sequence length while maintaining the ability to selectively store and retrieve information.

The architecture:

The key insight is competitive slot access: tokens compete to write into a bounded number of memory slots via softmax. This naturally implements a form of memory management where the most relevant information gets stored and stale information is gradually forgotten. Unlike unbounded linear attention states, the fixed slot count guarantees constant memory.

Reference: Peng et al., “ABC: Attention with Bounded-memory Control”, 2022. https://arxiv.org/abs/2110.02488

Constructors

Constructor Summary
ABCLayer(int sequenceLength, int modelDimension, int numSlots, int numHeads, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) Creates a new ABC (Attention with Bounded-memory Control) layer.

Properties

Property Summary
bool SupportsTraining { get; }
int HeadDimension { get; } Gets the dimension per head.
int ModelDimension { get; } Gets the model dimension.
int NumHeads { get; } Gets the number of attention heads.
int NumSlots { get; } Gets the number of memory slots per head.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Tensor<T> GetOutputProjectionWeights() Gets the output projection weights for external inspection.
Tensor<T> GetQueryWeights() Gets the query weights for external inspection.
Tensor<T> GetSlotKeys() Gets the slot key embeddings for external inspection.
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).