ExpectedCalibrationError<T>

Models & Types · AiDotNet.UncertaintyQuantification.Calibration

Computes the Expected Calibration Error (ECE) metric for evaluating probability calibration.

For Beginners

Expected Calibration Error (ECE) measures how well a model’s predicted probabilities match actual outcomes.

Imagine a weather forecaster who predicts “70% chance of rain”:

  • If it rains 70% of the time when they make this prediction, they’re well-calibrated (ECE near 0)
  • If it only rains 50% of the time, they’re overconfident (ECE is higher)

How ECE works:

  1. Group all predictions into bins by confidence (e.g., 0-10%, 10-20%, …, 90-100%)
  2. For each bin, compute:
  • Average predicted probability
  • Actual accuracy (fraction of correct predictions)
  1. ECE is the weighted average of the absolute differences

A well-calibrated model should have ECE < 0.05 (5%).

This is the gold standard metric for evaluating uncertainty quantification in classification tasks.

Constructors

Constructor Summary
ExpectedCalibrationError(int numBins) Initializes a new instance of the ExpectedCalibrationError class.

Methods

Method Summary
List<ValueTuple<T, T, int>> GetReliabilityDiagram(Vector<T> probabilities, Vector<int> predictions, Vector<int> trueLabels) Gets reliability diagram data for visualization.
T Compute(Vector<T> probabilities, Vector<int> predictions, Vector<int> trueLabels) Computes the Expected Calibration Error for predicted probabilities.