MixtureOfMemoriesLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers.SSM
Implements the Mixture of Memories (MoM) layer from Chou et al., 2025.
Inheritance: LayerBase<T> → MixtureOfMemoriesLayer<T>
Implements: IShapeContract
For Beginners
Think of this like having multiple filing cabinets (memories) instead of one:
Standard linear attention = one filing cabinet where all documents go. Over time, the cabinet gets cluttered and finding specific documents is hard.
MoM = multiple filing cabinets, each for different topics:
- A router (like a librarian) decides which cabinet to file each new document in (write routing)
- When you need information, the librarian checks relevant cabinets and combines results (read routing)
- Each cabinet has its own retention policy: some keep documents forever, others regularly clean out (forget gate)
This prevents unrelated information from interfering with each other, which is the main weakness of standard linear attention. The model learns to organize information across memories, much like a well-organized library system.
The number of memories M is a key hyperparameter:
- More memories = less interference, more capacity, but more parameters
- Fewer memories = simpler model, but more compression needed
- 4-8 memories is typically a good balance
How It Works
Standard linear attention maintains a single key-value memory state S. As the sequence grows, all information is compressed into this one matrix, leading to interference: unrelated key-value associations overwrite each other. MoM addresses this by maintaining multiple independent memory states (S_1, S_2, …, S_M) and using a learned router to selectively read from and write to them.
The architecture at each timestep t:
The routing mechanism is the key innovation: by directing different tokens to different memories, MoM prevents interference between unrelated information. This is analogous to how Mixture of Experts (MoE) routes tokens to different expert networks, but applied to the memory states of a linear attention model.
The forget gate per memory allows selective retention: some memories can maintain long-term state (high g) while others are more transient (low g), naturally specializing into different timescales.
Reference: Chou et al., “MoM: Mixture of Memories for Linear Sequence Modeling”, 2025.
https://arxiv.org/abs/2502.13685
Constructors
| Constructor | Summary |
|---|---|
MixtureOfMemoriesLayer(int sequenceLength, int modelDimension, int numHeads, int numMemories, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) |
Creates a new Mixture of Memories (MoM) layer. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int HeadDimension { get; } |
Gets the dimension per head. |
int ModelDimension { get; } |
Gets the model dimension. |
int NumHeads { get; } |
Gets the number of heads. |
int NumMemories { get; } |
Gets the number of memory states. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
Tensor<T> GetOutputProjectionWeights() |
Gets the output projection weights for external inspection. |
Tensor<T> GetQueryWeights() |
Gets the query weights for external inspection. |
Tensor<T> GetReadRouterWeights() |
Gets the read router weights for external inspection. |
Tensor<T> GetWriteRouterWeights() |
Gets the write router weights for external inspection. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |