GIFTEvalBenchmark<T>

Models & Types · AiDotNet.Finance.Evaluation

GIFT-Eval benchmark implementation for standardized evaluation of time series foundation models.

For Beginners

GIFT-Eval (General Time Series Forecasting Model Evaluation) is the standard benchmark used by the research community to compare foundation models. It uses two key metrics:

  • MASE (Mean Absolute Scaled Error): Measures point forecast accuracy relative

to a naive baseline. A MASE of 1.0 means the model performs the same as naive forecasting; values below 1.0 mean it’s better.

  • CRPS (Continuous Ranked Probability Score): Measures probabilistic forecast

quality by comparing predicted distributions to actual values. Lower is better.

How It Works

Reference: Jain et al., “GIFT-Eval: A Benchmark for General Time Series Forecasting Model Evaluation”, 2024. https://arxiv.org/abs/2410.10393

Leaderboard: https://huggingface.co/spaces/Salesforce/GIFT-Eval

Constructors

Constructor Summary
GIFTEvalBenchmark()

Methods

Method Summary
Dictionary<string, double> RunBenchmark(ITimeSeriesFoundationModel<T> model, IReadOnlyList<Tensor<T>> testSeries, IReadOnlyList<int> contextLengths, IReadOnlyList<int> seasonalPeriods, double[] quantileLevels) Runs a comprehensive GIFT-Eval style evaluation on a foundation model.
double ComputeCRPS(IReadOnlyDictionary<double, Tensor<T>> quantileForecasts, Tensor<T> actuals) Computes the Continuous Ranked Probability Score (CRPS) for quantile/probabilistic forecasts.
double ComputeMASE(Tensor<T> predictions, Tensor<T> actuals, Tensor<T> historicalData, int seasonalPeriod) Computes the Mean Absolute Scaled Error (MASE) for point forecasts.
double ComputeQuantileLoss(Tensor<T> predictions, Tensor<T> actuals, double quantileLevel) Computes the quantile loss (pinball loss) for a specific quantile level.