ExpectedCalibrationError<T>
Models & Types · AiDotNet.UncertaintyQuantification.Calibration
Computes the Expected Calibration Error (ECE) metric for evaluating probability calibration.
For Beginners
Expected Calibration Error (ECE) measures how well a model’s predicted probabilities match actual outcomes.
Imagine a weather forecaster who predicts “70% chance of rain”:
- If it rains 70% of the time when they make this prediction, they’re well-calibrated (ECE near 0)
- If it only rains 50% of the time, they’re overconfident (ECE is higher)
How ECE works:
- Group all predictions into bins by confidence (e.g., 0-10%, 10-20%, …, 90-100%)
- For each bin, compute:
- Average predicted probability
- Actual accuracy (fraction of correct predictions)
- ECE is the weighted average of the absolute differences
A well-calibrated model should have ECE < 0.05 (5%).
This is the gold standard metric for evaluating uncertainty quantification in classification tasks.
Constructors
| Constructor | Summary |
|---|---|
ExpectedCalibrationError(int numBins) |
Initializes a new instance of the ExpectedCalibrationError class. |
Methods
| Method | Summary |
|---|---|
List<ValueTuple<T, T, int>> GetReliabilityDiagram(Vector<T> probabilities, Vector<int> predictions, Vector<int> trueLabels) |
Gets reliability diagram data for visualization. |
T Compute(Vector<T> probabilities, Vector<int> predictions, Vector<int> trueLabels) |
Computes the Expected Calibration Error for predicted probabilities. |