TimeMoEBlockLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

A single Time-MoE transformer block: multi-head self-attention + Mixture-of-Experts FFN, each wrapped in pre-norm + residual. Per Shi et al. 2024 “Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts”.

Inheritance: LayerBase<T> → TimeMoEBlockLayer<T>

Implements: IShapeContract

How It Works

Forward sequence (pre-norm, GPT-style):

  • norm1(input) → self-attention → residual add with input → x
  • norm2(x) → MoE-FFN (top-k expert dispatch) → residual add with x → output

The MoE routes each token (each row of the flattened [B·numPatches, hiddenDim] tensor) independently through top-k experts, per paper. Each expert is a 2-layer Dense FFN with GELU (hiddenDim → intermediateSize → hiddenDim). Load-balancing is enabled with weight 0.01.

Constructors

Constructor Summary
TimeMoEBlockLayer(int hiddenDim, int numHeads, int intermediateSize, int numExperts, int topK) Initializes a new TimeMoEBlockLayer.

Properties

Property Summary
bool SupportsTraining { get; }

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
T GetAuxiliaryLoss() Surfaces the MoE router’s load-balancing auxiliary loss for the current (most recent) forward pass.
void ResetState()
void UpdateParameters(T learningRate)