HedgehogLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers.SSM

Implements the Hedgehog layer from “The Hedgehog and the Porcupine: Expressive Linear Attentions with Softmax Mimicry” (Zhang et al., 2024, ICLR 2024).

Inheritance: LayerBase<T> → HedgehogLayer<T>

Implements: IShapeContract

For Beginners

Hedgehog makes linear attention much better by learning HOW to pay attention.

In standard linear attention, there’s a mathematical trick (feature map) that converts the expensive softmax operation into a cheaper form. But the standard tricks (like adding 1 and clipping negatives) are crude approximations. It’s like trying to draw a circle using only straight lines – you can approximate it, but it’s never quite right.

Hedgehog says: “Instead of using a fixed approximation, let’s LEARN the best feature map from data.” It uses a small neural network (just two layers) that is trained alongside the main model. This learned feature map can capture much more nuanced attention patterns than any fixed formula.

The result: Hedgehog gets close to the quality of full softmax attention while keeping the speed advantage of linear attention (O(n) instead of O(n^2)).

Think of it this way:

  • Standard linear attention: “I’ll use this simple formula to decide what’s important”
  • Hedgehog: “I’ll learn the BEST formula for deciding what’s important from the data itself”

How It Works

Hedgehog replaces the fixed feature map in linear attention with a small trainable MLP that learns to approximate softmax attention from data. Standard linear attention uses simple feature maps like ELU(x)+1 or polynomial expansions, which poorly approximate softmax and lead to degraded quality. Hedgehog instead trains the feature map end-to-end, achieving much better softmax approximation.

The architecture:

The key insight is that softmax attention can be decomposed as a kernel: softmax(QK^T) = phi(Q) phi(K)^T for some (unknown) feature map phi. Rather than using a fixed approximation, Hedgehog learns phi directly. The MLP is small (typically 64 hidden units) so the overhead is minimal, but the quality improvement over fixed feature maps is substantial – closing most of the gap to full softmax attention.

Reference: Zhang et al., “The Hedgehog and the Porcupine: Expressive Linear Attentions with Softmax Mimicry”, ICLR 2024. https://arxiv.org/abs/2402.04347

Constructors

Constructor Summary
HedgehogLayer(int sequenceLength, int modelDimension, int numHeads, int featureMapHiddenDim, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy)

Properties

Property Summary
bool SupportsTraining { get; }
int FeatureMapHiddenDim { get; } Gets the hidden dimension of the feature map MLP.
int HeadDimension { get; } Gets the dimension per head.
int ModelDimension { get; } Gets the model dimension.
int NumHeads { get; } Gets the number of attention heads.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Tensor<T> GetFeatureMapW1() Gets the feature map W1 weights for external inspection.
Tensor<T> GetFeatureMapW2() Gets the feature map W2 weights for external inspection.
Tensor<T> GetOutputProjectionWeights() Gets the output projection weights for external inspection.
Tensor<T> GetQueryWeights() Gets the query weights for external inspection.
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).