RodimusLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers.SSM

Implements the Rodimus layer from “Rodimus: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentions” (He et al., 2025).

Inheritance: LayerBase<T> → RodimusLayer<T>

Implements: IShapeContract

For Beginners

Rodimus is a smart attention mechanism that can adjust its “focus level” depending on the input.

Think of reading a textbook:

  • Sometimes you need to focus sharply on one specific definition (low temperature = very selective)
  • Other times you need to understand the general theme of a paragraph (high temperature = broad focus)
  • A good reader adjusts their focus level based on what they’re reading

Rodimus does exactly this:

  • It learns a “temperature” for each position that controls focus sharpness
  • Low temperature = laser focus on the most relevant information
  • High temperature = broad survey of all available information
  • The temperature is “data-dependent” meaning it adjusts based on the input itself

Combined with a gated recurrence (like LSTM but for a matrix-valued state), this gives Rodimus the quality of Transformer attention with the efficiency of linear recurrence. The gated recurrence maintains a running state matrix that gets selectively updated at each step, avoiding the O(n^2) cost of full attention.

How It Works

Rodimus combines a data-dependent tempered selection mechanism with gated linear recurrence to achieve both high quality and linear-time efficiency. The key innovation is using a temperature-scaled softmax for selective state updates, allowing the model to dynamically control the sharpness of its attention/selection mechanism.

The architecture:

The temperature parameter tau is crucial: it controls the “sharpness” of attention.

  • Low temperature (tau near 0): Very selective, focuses on the best-matching key (like argmax)
  • High temperature (tau near infinity): Uniform attention, treats all keys equally
  • Data-dependent: The model learns when to be selective vs. when to spread attention

This allows Rodimus to adaptively decide: “Should I focus sharply on one specific key-value pair (low temp), or should I aggregate broadly (high temp)?” This breaks the typical accuracy-efficiency trade-off because the model can be precise when needed and efficient otherwise.

Reference: He et al., “Rodimus: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentions”, 2025. https://arxiv.org/abs/2410.06577

Constructors

Constructor Summary
RodimusLayer(int sequenceLength, int modelDimension, int numHeads, double temperature, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) Creates a new Rodimus layer with data-dependent tempered selection.

Properties

Property Summary
bool SupportsTraining { get; }
double BaseTemperature { get; } Gets the base temperature value.
int HeadDimension { get; } Gets the dimension per head.
int ModelDimension { get; } Gets the model dimension.
int NumHeads { get; } Gets the number of attention heads.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Tensor<T> GetOutputProjectionWeights() Gets the output projection weights for external inspection.
Tensor<T> GetQueryWeights() Gets the query weights for external inspection.
Tensor<T> GetTemperatureWeights() Gets the temperature weights for external inspection.
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).