MEGALayer<T>
Layers · AiDotNet.NeuralNetworks.Layers.SSM
Implements the MEGA (Moving Average Equipped Gated Attention) layer from Ma et al., 2023.
Inheritance: LayerBase<T> → MEGALayer<T>
Implements: IShapeContract
For Beginners
MEGA is like having two complementary systems working together:
- The EMA is like a set of “smoothing filters” applied to the input sequence. Imagine running
your finger along a line in a graph to smooth out bumps – each dimension has its own smoothing strength. Some dimensions smooth heavily (capturing broad trends), others barely smooth at all (preserving sharp details).
- The attention mechanism then looks at these smoothed representations and decides which positions
are important for each output. Because the input has already been smoothed, the attention can focus on content similarity rather than having to also learn positional patterns.
Together, EMA handles “where to look” (position-aware) and attention handles “what’s relevant” (content-aware), dividing the labor efficiently.
How It Works
MEGA combines an exponential moving average (EMA) with gated single-head attention to capture both position-aware local smoothing and content-based global mixing. It achieves strong results on sequence modeling benchmarks while being significantly more efficient than multi-head attention.
The architecture:
The EMA acts as a learned positional prior: it smooths the input with different decay rates per dimension, so some dimensions capture very local context (fast decay) while others retain long-range information (slow decay). The attention mechanism then operates on these position-aware representations, making it easier to learn both local and global dependencies.
Reference: Ma et al., “MEGA: Moving Average Equipped Gated Attention”, ICLR 2023.
https://arxiv.org/abs/2209.10655
Constructors
| Constructor | Summary |
|---|---|
MEGALayer(int sequenceLength, int modelDimension, int numHeads, int emaDimension, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) |
Creates a new MEGA (Moving Average Equipped Gated Attention) layer. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int EmaDimension { get; } |
Gets the EMA dimension. |
int HeadDimension { get; } |
Gets the dimension per head. |
int ModelDimension { get; } |
Gets the model dimension. |
int NumHeads { get; } |
Gets the number of attention heads. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns the complete convention-enumerated trainable tensor surface. |
Tensor<T> GetEmaAlphaLogit() |
Gets the EMA alpha logit parameters for external inspection. |
Tensor<T> GetOutputProjectionWeights() |
Gets the output projection weights for external inspection. |
Tensor<T> GetQueryWeights() |
Gets the query weights for external inspection. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
|
void UpdateParameters(T learningRate) |