MegalodonLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers.SSM

Implements the Megalodon layer from “Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length” (Ma et al., 2024, arXiv:2404.08801).

Inheritance: LayerBase<T> → MegalodonLayer<T>

Implements: IShapeContract

For Beginners

Megalodon is like having a bank of tuning forks that each ring at different frequencies.

Imagine you are analyzing a complex audio signal:

  • A standard EMA is like a simple echo that fades over time – it can only remember things

that happened recently, and the memory just gets quieter (exponential decay).

  • Megalodon’s CEMA is like a set of tuning forks. Each fork vibrates at its own frequency.

When you tap a tuning fork (give it input), it rings and slowly fades, but it OSCILLATES while fading. This means it naturally picks up patterns that repeat at that frequency.

The “complex” in CEMA means each decay coefficient has two parts:

  • A real part: controls how fast the oscillation fades (the damping)
  • An imaginary part: controls how fast it oscillates (the frequency)

Together, many CEMA dimensions with different frequencies act like a Fourier-style analyzer that decomposes the input into its frequency components, all within linear O(n) complexity.

The timestep normalization prevents the “tuning forks” from getting too loud over very long sequences, which is why Megalodon can handle unlimited context length.

How It Works

Megalodon extends the MEGA (Moving Average Equipped Gated Attention) architecture with two key innovations that enable unlimited context length:

The CEMA mechanism is the core innovation. Standard EMA uses real-valued decay coefficients, which can only model exponential decay. By making alpha complex-valued, the state evolves as a damped oscillation: |alpha| controls the decay rate, while arg(alpha) controls the oscillation frequency. This gives each EMA dimension a unique frequency response, enabling the model to selectively attend to different periodicities in the input.

Timestep normalization is critical for unlimited context. Without it, the accumulated EMA state grows with sequence length, causing numerical instability. By normalizing per-timestep, Megalodon maintains stable representations regardless of context length.

Reference: Ma et al., “Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length”, arXiv:2404.08801, 2024.

Constructors

Constructor Summary
MegalodonLayer(int sequenceLength, int modelDimension, int numHeads, int emaDimension, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy)

Properties

Property Summary
bool SupportsTraining { get; }
int EmaDimension { get; } Gets the EMA state dimension (number of complex EMA channels).
int HeadDimension { get; } Gets the dimension per attention head.
int ModelDimension { get; } Gets the model dimension.
int NumHeads { get; } Gets the number of attention heads.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Tensor<T> GetEmaAlphaImag() Gets the CEMA alpha coefficients (imaginary part) for external inspection.
Tensor<T> GetEmaAlphaReal() Gets the CEMA alpha coefficients (real part) for external inspection.
Tensor<T> GetOutputProjectionWeights() Gets the output projection weights for external inspection.
Tensor<T> GetQueryWeights() Gets the query weights for external inspection.
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).