MEGALayer<T>

Layers · AiDotNet.NeuralNetworks.Layers.SSM

Implements the MEGA (Moving Average Equipped Gated Attention) layer from Ma et al., 2023.

Inheritance: LayerBase<T> → MEGALayer<T>

Implements: IShapeContract

For Beginners

MEGA is like having two complementary systems working together:

  1. The EMA is like a set of “smoothing filters” applied to the input sequence. Imagine running

your finger along a line in a graph to smooth out bumps – each dimension has its own smoothing strength. Some dimensions smooth heavily (capturing broad trends), others barely smooth at all (preserving sharp details).

  1. The attention mechanism then looks at these smoothed representations and decides which positions

are important for each output. Because the input has already been smoothed, the attention can focus on content similarity rather than having to also learn positional patterns.

Together, EMA handles “where to look” (position-aware) and attention handles “what’s relevant” (content-aware), dividing the labor efficiently.

How It Works

MEGA combines an exponential moving average (EMA) with gated single-head attention to capture both position-aware local smoothing and content-based global mixing. It achieves strong results on sequence modeling benchmarks while being significantly more efficient than multi-head attention.

The architecture:

The EMA acts as a learned positional prior: it smooths the input with different decay rates per dimension, so some dimensions capture very local context (fast decay) while others retain long-range information (slow decay). The attention mechanism then operates on these position-aware representations, making it easier to learn both local and global dependencies.

Reference: Ma et al., “MEGA: Moving Average Equipped Gated Attention”, ICLR 2023. https://arxiv.org/abs/2209.10655

Constructors

Constructor Summary
MEGALayer(int sequenceLength, int modelDimension, int numHeads, int emaDimension, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) Creates a new MEGA (Moving Average Equipped Gated Attention) layer.

Properties

Property Summary
bool SupportsTraining { get; }
int EmaDimension { get; } Gets the EMA dimension.
int HeadDimension { get; } Gets the dimension per head.
int ModelDimension { get; } Gets the model dimension.
int NumHeads { get; } Gets the number of attention heads.

Methods

Method Summary
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns the complete convention-enumerated trainable tensor surface.
Tensor<T> GetEmaAlphaLogit() Gets the EMA alpha logit parameters for external inspection.
Tensor<T> GetOutputProjectionWeights() Gets the output projection weights for external inspection.
Tensor<T> GetQueryWeights() Gets the query weights for external inspection.
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters)
void UpdateParameters(T learningRate)