TimeMoEBlockLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
A single Time-MoE transformer block: multi-head self-attention + Mixture-of-Experts FFN, each wrapped in pre-norm + residual. Per Shi et al. 2024 “Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts”.
Inheritance: LayerBase<T> → TimeMoEBlockLayer<T>
Implements: IShapeContract
How It Works
Forward sequence (pre-norm, GPT-style):
- norm1(input) → self-attention → residual add with input → x
- norm2(x) → MoE-FFN (top-k expert dispatch) → residual add with x → output
The MoE routes each token (each row of the flattened [B·numPatches, hiddenDim] tensor) independently through top-k experts, per paper. Each expert is a 2-layer Dense FFN with GELU (hiddenDim → intermediateSize → hiddenDim). Load-balancing is enabled with weight 0.01.
Constructors
| Constructor | Summary |
|---|---|
TimeMoEBlockLayer(int hiddenDim, int numHeads, int intermediateSize, int numExperts, int topK) |
Initializes a new TimeMoEBlockLayer. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
T GetAuxiliaryLoss() |
Surfaces the MoE router’s load-balancing auxiliary loss for the current (most recent) forward pass. |
void ResetState() |
|
void UpdateParameters(T learningRate) |