ProbabilityCalibrator<T>

Models & Types · AiDotNet.Evaluation.Calibration

Calibrates model outputs to produce reliable probability estimates.

For Beginners

Calibration is like adjusting a thermometer. Your model might always say “90 degrees” when the true temperature is 85. Calibration learns this systematic error and corrects for it.

Available methods:

  • Platt Scaling: Fits a logistic curve - good for SVMs and many classifiers
  • Isotonic Regression: Non-parametric curve fitting - more flexible but needs more data
  • Temperature Scaling: Simple division - popular for neural networks
  • Histogram Binning: Averages within bins - simple and interpretable
  • Beta Calibration: Fits beta distribution - good for bounded outputs

Example usage:

  1. Train your model and get predicted probabilities
  2. Fit the calibrator on a held-out validation set
  3. Use calibrator to transform all future predictions

How It Works

Probability calibration transforms raw model scores into well-calibrated probabilities. A well-calibrated model means: if you see 1000 predictions of “70% probability”, about 700 of those should be positive outcomes.

Constructors

Constructor Summary
ProbabilityCalibrator(ProbabilityCalibratorOptions options) Initializes a new instance of the calibrator.

Properties

Property Summary
ProbabilityCalibrationMethod ProbabilityCalibrationMethod { get; } Gets the calibration method being used.

Methods

Method Summary
T GetTemperature() Gets the temperature parameter.
ValueTuple<T, T, T> GetBetaParameters() Gets the beta calibration parameters (a, b, c).
ValueTuple<T, T> GetPlattParameters() Gets the Platt scaling parameters (A, B).
ValueTuple<T[], T[], int[]> GetReliabilityDiagram(Vector<T> scores, Vector<T> labels, int numBins) Returns mean predictions, positive fractions and exact counts per uniform bin.
Vector<T> FitTransform(Vector<T> scores, Vector<T> labels) Fits the calibrator and transforms in one step.
Vector<T> Transform(Vector<T> scores) Transforms predicted scores into calibrated probabilities.
double ComputeBrierScore(Vector<T> scores, Vector<T> labels) Computes the mean squared difference between scores and labels.
double ComputeECE(Vector<T> scores, Vector<T> labels, int numBins) Computes the Expected Calibration Error (ECE).
double ComputeMCE(Vector<T> scores, Vector<T> labels, int numBins) Computes the largest absolute calibration error among non-empty bins.
void Fit(Vector<T> scores, Vector<T> labels) Fits the calibrator to predicted scores and true labels.