LonghornLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers.SSM
Implements the Longhorn layer from “Longhorn: State Space Models are Amortized Online Learners” (Liu et al., 2024).
Inheritance: LayerBase<T> → LonghornLayer<T>
Implements: IShapeContract
For Beginners
Longhorn is a sequence model that maintains an internal “memory” which works like a student learning from a stream of examples.
Imagine a student studying flashcards one at a time:
- Each flashcard has a “key” (the question) and a “value” (the answer)
- The student maintains a mental model (the state matrix S) that maps keys to values
- For each new flashcard, the student decides how much to “learn” from it (alpha)
- High alpha: “This is important, update my understanding significantly”
- Low alpha: “I already know this well, barely change my understanding”
- To answer a question (query), the student applies their mental model
The key difference from standard attention:
- Standard attention: Re-reads ALL flashcards every time (O(n^2) cost)
- Longhorn: Maintains a compressed summary that gets updated incrementally (O(n) cost)
This makes Longhorn much more efficient for long sequences while still capturing important patterns.
How It Works
Longhorn reinterprets state space model state updates as amortized online learning. Rather than treating the recurrent state as an opaque hidden state, Longhorn views it as a weight matrix of an online learner that is continuously updated to predict values from keys. The decay/forget gate acts as a data-dependent learning rate controlling how quickly the “learner” adapts to new observations.
The architecture:
The critical insight is that this state update rule is equivalent to an online learning rule:
- The state matrix S acts as the “model weights” of a linear predictor
- Each (k_t, v_t) pair is a new “training example”
- alpha_t is the “learning rate” that controls how much to update
- (1 - alpha_t) is the “retention rate” that controls how much to remember
- Querying S with q_t is equivalent to running inference on the online learner
Unlike GatedDeltaNet which uses a delta rule (error-corrective) update, Longhorn uses a simpler exponential moving average update. This means it does not check what the state already knows before writing – it simply blends the old state with the new outer product. Despite this simplicity, the online learning perspective enables principled initialization and understanding of the model’s behavior.
Reference: Liu et al., “Longhorn: State Space Models are Amortized Online Learners”, 2024.
https://arxiv.org/abs/2407.14207
Constructors
| Constructor | Summary |
|---|---|
LonghornLayer(int sequenceLength, int modelDimension, int numHeads, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) |
Creates a new Longhorn layer. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int HeadDimension { get; } |
Gets the dimension per head (modelDimension / numHeads). |
int ModelDimension { get; } |
Gets the model dimension (d_model). |
int NumHeads { get; } |
Gets the number of attention heads. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns the complete convention-enumerated trainable tensor surface. |
Tensor<T> GetAlphaWeights() |
Gets the alpha (learning rate gate) weights for external inspection or analysis. |
Tensor<T> GetOutputProjectionWeights() |
Gets the output projection weights for external inspection or analysis. |
Tensor<T> GetQueryWeights() |
Gets the query weights for external inspection or analysis. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
|
void UpdateParameters(T learningRate) |