KimiLinearAttentionLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers.SSM
Implements the Kimi KDA (Key-Value Driven Gated Linear Attention) layer from the “Kimi-VL Technical Report” (Kimi Team, 2025, arXiv:2510.26692).
Inheritance: LayerBase<T> → KimiLinearAttentionLayer<T>
Implements: IShapeContract
For Beginners
Kimi KDA is a way to read through a sequence and build up a memory of what you’ve seen, with a smart forgetting mechanism.
Think of it like taking notes while reading a book:
- At each word, you have a “key” (what topic this is about) and a “value” (what it says)
- The gate is like asking: “Does this new information FIT with what I already know?”
- If key and value are very aligned (k^T * v is large): “This is strong, clear info” -> gate near 1
- If they’re not aligned: “This is ambiguous or contradictory” -> gate lower
- When the gate is high: keep most of old notes (g * S) and add new info (k * v^T)
- When the gate is low: forget more old notes, making room for new patterns
This is different from other approaches that gate based on the INPUT position:
- Position-based gate: “I’m at word 50, so I should forget some old stuff” (doesn’t know WHAT to forget)
- KV-driven gate: “This new key-value pair conflicts with stored patterns, so selectively forget” (content-aware)
The result is more intelligent memory management that adapts to the actual information content.
How It Works
Kimi KDA is a gated linear attention mechanism where the gate is computed from the interaction between keys and values rather than from a separate learned projection. The gate signal g_t = sigma(k_t^T * v_t + bias) captures whether the current key-value pair contains information that conflicts with or reinforces the current state. This KV-driven gating replaces the typical input-driven gating found in other linear attention variants.
The architecture:
The key insight is that the gate should reflect the CONTENT of what’s being stored, not just where in the sequence we are. When k_t and v_t are highly aligned (high dot product), the information is coherent and can be safely accumulated. When they conflict with stored patterns, the gate should allow partial forgetting. This KV-driven approach automatically detects when new information should overwrite old information.
Reference: Kimi Team, “Kimi-VL Technical Report”, 2025.
https://arxiv.org/abs/2510.26692
Constructors
| Constructor | Summary |
|---|---|
KimiLinearAttentionLayer(int sequenceLength, int modelDimension, int numHeads, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) |
Creates a new Kimi KDA (Key-Value Driven Gated Linear Attention) layer. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int HeadDimension { get; } |
Gets the dimension per head. |
int ModelDimension { get; } |
Gets the model dimension. |
int NumHeads { get; } |
Gets the number of heads. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
Tensor<T> GetOutputProjectionWeights() |
Gets the output projection weights for external inspection. |
Tensor<T> GetQueryWeights() |
Gets the query weights for external inspection. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |