GIFTEvalBenchmark<T>
Models & Types · AiDotNet.Finance.Evaluation
GIFT-Eval benchmark implementation for standardized evaluation of time series foundation models.
For Beginners
GIFT-Eval (General Time Series Forecasting Model Evaluation) is the standard benchmark used by the research community to compare foundation models. It uses two key metrics:
- MASE (Mean Absolute Scaled Error): Measures point forecast accuracy relative
to a naive baseline. A MASE of 1.0 means the model performs the same as naive forecasting; values below 1.0 mean it’s better.
- CRPS (Continuous Ranked Probability Score): Measures probabilistic forecast
quality by comparing predicted distributions to actual values. Lower is better.
How It Works
Reference: Jain et al., “GIFT-Eval: A Benchmark for General Time Series Forecasting
Model Evaluation”, 2024. https://arxiv.org/abs/2410.10393
Leaderboard: https://huggingface.co/spaces/Salesforce/GIFT-Eval
Constructors
| Constructor | Summary |
|---|---|
GIFTEvalBenchmark() |
Methods
| Method | Summary |
|---|---|
Dictionary<string, double> RunBenchmark(ITimeSeriesFoundationModel<T> model, IReadOnlyList<Tensor<T>> testSeries, IReadOnlyList<int> contextLengths, IReadOnlyList<int> seasonalPeriods, double[] quantileLevels) |
Runs a comprehensive GIFT-Eval style evaluation on a foundation model. |
double ComputeCRPS(IReadOnlyDictionary<double, Tensor<T>> quantileForecasts, Tensor<T> actuals) |
Computes the Continuous Ranked Probability Score (CRPS) for quantile/probabilistic forecasts. |
double ComputeMASE(Tensor<T> predictions, Tensor<T> actuals, Tensor<T> historicalData, int seasonalPeriod) |
Computes the Mean Absolute Scaled Error (MASE) for point forecasts. |
double ComputeQuantileLoss(Tensor<T> predictions, Tensor<T> actuals, double quantileLevel) |
Computes the quantile loss (pinball loss) for a specific quantile level. |