RodimusLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers.SSM
Implements the Rodimus layer from “Rodimus: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentions” (He et al., 2025).
Inheritance: LayerBase<T> → RodimusLayer<T>
Implements: IShapeContract
For Beginners
Rodimus is a smart attention mechanism that can adjust its “focus level” depending on the input.
Think of reading a textbook:
- Sometimes you need to focus sharply on one specific definition (low temperature = very selective)
- Other times you need to understand the general theme of a paragraph (high temperature = broad focus)
- A good reader adjusts their focus level based on what they’re reading
Rodimus does exactly this:
- It learns a “temperature” for each position that controls focus sharpness
- Low temperature = laser focus on the most relevant information
- High temperature = broad survey of all available information
- The temperature is “data-dependent” meaning it adjusts based on the input itself
Combined with a gated recurrence (like LSTM but for a matrix-valued state), this gives Rodimus the quality of Transformer attention with the efficiency of linear recurrence. The gated recurrence maintains a running state matrix that gets selectively updated at each step, avoiding the O(n^2) cost of full attention.
How It Works
Rodimus combines a data-dependent tempered selection mechanism with gated linear recurrence to achieve both high quality and linear-time efficiency. The key innovation is using a temperature-scaled softmax for selective state updates, allowing the model to dynamically control the sharpness of its attention/selection mechanism.
The architecture:
The temperature parameter tau is crucial: it controls the “sharpness” of attention.
- Low temperature (tau near 0): Very selective, focuses on the best-matching key (like argmax)
- High temperature (tau near infinity): Uniform attention, treats all keys equally
- Data-dependent: The model learns when to be selective vs. when to spread attention
This allows Rodimus to adaptively decide: “Should I focus sharply on one specific key-value pair (low temp), or should I aggregate broadly (high temp)?” This breaks the typical accuracy-efficiency trade-off because the model can be precise when needed and efficient otherwise.
Reference: He et al., “Rodimus: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentions”, 2025.
https://arxiv.org/abs/2410.06577
Constructors
| Constructor | Summary |
|---|---|
RodimusLayer(int sequenceLength, int modelDimension, int numHeads, double temperature, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) |
Creates a new Rodimus layer with data-dependent tempered selection. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
double BaseTemperature { get; } |
Gets the base temperature value. |
int HeadDimension { get; } |
Gets the dimension per head. |
int ModelDimension { get; } |
Gets the model dimension. |
int NumHeads { get; } |
Gets the number of attention heads. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
Tensor<T> GetOutputProjectionWeights() |
Gets the output projection weights for external inspection. |
Tensor<T> GetQueryWeights() |
Gets the query weights for external inspection. |
Tensor<T> GetTemperatureWeights() |
Gets the temperature weights for external inspection. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |