TemporalFusionTransformer<T>

Models & Types · AiDotNet.TimeSeries

Implements the Temporal Fusion Transformer (TFT) per Lim et al. (2021). Architecture retained here: input embedding → variable selection (softmax-weighted feature combination) → static-enrichment GRN → interpretable multi-head attention → post-attention gated skip (GRN) → forecast head.

Inheritance: TimeSeriesModelBase<T> → TemporalFusionTransformer<T>

Implements: IGeneratedParameterRegistrar<T>

How It Works

Tape-based training. The entire forward pass (ForwardBatch) is expressed with batched Engine.Tensor* ops, so a GradientTape produces the whole backward pass automatically (no hand-derived gradients) and every op is GPU-dispatchable. Windows of the z-normalized series are stacked into a [B, L] batch and the full H-step horizon is supervised per sample with the pinball / quantile loss summed over every configured QuantileLevels level, so the model learns a genuine predictive spread (see PredictQuantiles); the point forecast is the median-level head.

Simplifications vs the full paper (documented per the tape-conversion campaign): (1) The sequential LSTM encoder-decoder is replaced by additive sinusoidal positional encodings feeding the self-attention block (a recurrent scan differentiates poorly on the tape and does not batch; positional encodings + attention recover temporal order). (2) For univariate series (the only exogenous inputs available here) variable selection reduces to a learned per-channel softmax gate over the embedded features rather than selection across multiple exogenous variables. The GRN, variable-selection gating, and interpretable multi-head attention — TFT’s defining components — are all retained and fully differentiable.

Example

using AiDotNet;
using AiDotNet.Data.Loaders;
using AiDotNet.TimeSeries;
using AiDotNet.Tensors.LinearAlgebra;

double[] series =
{
    120, 135, 148, 160, 155, 170, 180, 195, 210, 198, 220, 235,
    140, 155, 165, 178, 172, 190, 200, 215, 230, 218, 245, 260
};
var x = new Matrix<double>(series.Length, 1);
for (int i = 0; i < series.Length; i++) x[i, 0] = i;

var result = await new AiModelBuilder<double, Matrix<double>, Vector<double>>()
    .ConfigureModel(new TemporalFusionTransformer<double>())
    .ConfigureDataLoader(DataLoaders.FromMatrixVector(x, new Vector<double>(series)))
    .BuildAsync();

var forecast = result.Predict(new Matrix<double>(6, 1));
Console.WriteLine($"TemporalFusionTransformer: forecast {forecast.Length} steps.");

Constructors

Constructor Summary
TemporalFusionTransformer(TemporalFusionTransformerOptions<T> options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T PredictSingle(Vector<T> input)
Vector<T> ForecastHorizon(Vector<T> input) Multi-horizon forecast for a single lookback window (H denormalized future values).
Vector<T> Predict(Matrix<T> input)
Vector<T> PredictQuantiles(Vector<T> input) Returns per-quantile forecasts as a flat vector laid out quantile-major: [q0_h0..q0_h(H-1), q1_h0..q1_h(H-1), ...] (index = qi*ForecastHorizon + h), in the same order as QuantileLevels.