Evaluation

179 public types in this namespace, organized by kind.

Models & Types (125)

Type Summary
AUCPRMetric<T> Computes Area Under the Precision-Recall Curve (AUC-PR).
AUCROCMetric<T> Computes Area Under the ROC Curve (AUC-ROC): measures discrimination ability.
AccuracyMetric<T> Computes classification accuracy: the proportion of correct predictions.
AdjustedR2Metric<T> Computes Adjusted R² Score: R² adjusted for the number of predictors in the model.
BalancedAccuracyMetric<T> Computes balanced accuracy: the average recall across all classes.
BalancedErrorRateMetric<T> Computes Balanced Error Rate (BER): average of FNR and FPR.
BlockedKFoldStrategy<T> Blocked K-Fold: K-Fold with temporal blocking (gap) between train and validation.
BootstrapStrategy<T> Bootstrap Cross-Validation: uses bootstrap sampling (sampling with replacement) for validation.
BootstrapTest<T> Bootstrap-based hypothesis test for comparing two models.
BrierScoreMetric<T> Computes Brier Score: mean squared error of probability predictions.
CRPSMetric<T> Continuous Ranked Probability Score metric for probabilistic predictions.
CalibrationBin<T> A single bin in the calibration curve.
CalibrationEngine<T> Engine for analyzing and improving probability calibration of classifiers.
CalibrationResult<T> Results from calibration analysis.
ClassificationResults<T> Classification-specific results container.
CochranQTest<T> Cochran’s Q test for comparing multiple classifiers on the same dataset.
CohensKappaMetric<T> Computes Cohen’s Kappa: agreement measure that accounts for chance agreement.
CrossValidationEngine<T> Engine for executing cross-validation with various strategies and aggregating results.
CrossValidationResult<T> Aggregated results from cross-validation across all folds.
DatasetStatistics<T> Statistics about the dataset used for evaluation.
DeLongTest<T> DeLong’s test for comparing two ROC curves.
DiagnosticOddsRatioMetric<T> Computes Diagnostic Odds Ratio: (TP × TN) / (FP × FN).
ErrorRateMetric<T> Computes Error Rate (Misclassification Rate): 1 - Accuracy.
EvaluationReport<T> Comprehensive evaluation report containing all computed metrics and analysis results.
EvaluationWarning Warning generated during evaluation.
ExplainedVarianceMetric<T> Computes Explained Variance Score: proportion of variance in target explained by predictions.
F1ScoreMetric<T> Computes F1 score: the harmonic mean of precision and recall.
FBetaScoreMetric<T> Computes F-beta score: weighted harmonic mean of precision and recall.
FairnessEngine<T> Engine for analyzing fairness and bias in machine learning models.
FairnessResult<T> Results from fairness analysis.
FalseDiscoveryRateMetric<T> Computes False Discovery Rate (FDR): FP / (FP + TP) = 1 - Precision.
FalseNegativeRateMetric<T> Computes False Negative Rate (FNR): proportion of actual positives incorrectly predicted as negative.
FalseOmissionRateMetric<T> Computes False Omission Rate (FOR): FN / (FN + TN) = 1 - NPV.
FalsePositiveRateMetric<T> Computes False Positive Rate (FPR): proportion of actual negatives incorrectly predicted as positive.
FeatureStatistics<T> Statistics for a single feature.
FoldResult<T> Results from a single cross-validation fold.
FowlkesMallowsMetric<T> Computes the Fowlkes-Mallows Index: geometric mean of precision and recall.
FriedmanTest<T> Friedman test: non-parametric test for comparing multiple classifiers across multiple datasets.
GiniCoefficientMetric<T> Computes Gini Coefficient (normalized): 2 * AUC - 1, measures discriminative ability.
GroupFairnessMetrics Fairness metrics for a specific group.
GroupKFoldStrategy<T> Group K-Fold: K-Fold that keeps related samples (groups) together in the same fold.
HammingLossMetric<T> Computes Hamming Loss: fraction of labels that are incorrectly predicted.
HingeLossMetric<T> Computes Hinge Loss: loss function used by Support Vector Machines (SVM).
HuberLossMetric<T> Computes Huber Loss: a robust loss function that is less sensitive to outliers than MSE.
InformednessMetric<T> Computes Informedness (Youden’s J statistic): TPR + TNR - 1 = Recall + Specificity - 1.
JaccardScoreMetric<T> Computes Jaccard Score (Intersection over Union): measures similarity between prediction and actual sets.
KFoldStrategy<T> K-Fold cross-validation: splits data into K equal-sized folds, using each as validation once.
KruskalWallisTest<T> Kruskal-Wallis H test for comparing multiple independent groups.
LearningCurveEngine<T> Engine for generating learning curves: how model performance changes with training set size.
LearningCurveResult<T> Results from learning curve analysis.
LeaveOneOutStrategy<T> Leave-One-Out Cross-Validation (LOOCV): each sample is used once as validation.
LeavePOutStrategy<T> Leave-P-Out cross-validation: train on N-P samples, validate on P samples for all combinations.
LeveneTest<T> Levene’s test for equality of variances across groups.
LogCoshLossMetric<T> Computes Log-Cosh Loss: mean of log(cosh(y - ŷ)).
LogLossMetric<T> Computes Log Loss (Cross-Entropy Loss): a probabilistic measure that penalizes confident wrong predictions.
LogScoreMetric<T> Logarithmic scoring metric for probabilistic predictions (Negative Log Likelihood).
MAEMetric<T> Computes Mean Absolute Error (MAE): average absolute difference between predictions and actuals.
MAPEMetric<T> Computes Mean Absolute Percentage Error (MAPE): average percentage error.
MASEMetric<T> Computes Mean Absolute Scaled Error (MASE): scale-independent measure for time series.
MASESeasonalMetric<T> Computes Seasonal MASE: Mean Absolute Scaled Error with explicit seasonal comparison.
MSEMetric<T> Computes Mean Squared Error (MSE): average squared difference between predictions and actuals.
MarkednessMetric<T> Computes Markedness: PPV + NPV - 1 = Precision + NPV - 1.
MatthewsCorrelationCoefficientMetric<T> Computes Matthews Correlation Coefficient (MCC): a balanced measure for binary and multi-class classification.
MaxErrorMetric<T> Computes Maximum Error: the worst-case absolute prediction error.
McNemarTest<T> McNemar’s test: compares the performance of two binary classifiers on the same dataset.
MeanAbsolutePercentageErrorMetric<T> Computes Mean Absolute Percentage Error with configurable handling of zeros.
MeanBiasErrorMetric<T> Computes Mean Bias Error (MBE): average signed error showing systematic over/under-prediction.
MeanDirectionalAccuracyMetric<T> Computes Mean Directional Accuracy (MDA): fraction of correctly predicted directions.
MeanSquaredLogErrorMetric<T> Computes Mean Squared Logarithmic Error (MSLE): squared version of RMSLE.
MedianAbsoluteErrorMetric<T> Computes Median Absolute Error (MedAE): the median of all absolute errors.
MetricCollection<T> A collection of metrics, providing easy access by name and category.
MetricComputationEngine<T> Core engine for computing evaluation metrics across all task types.
MetricWithCI<T> Represents a metric value with optional confidence interval and metadata.
ModelComparisonReport<T> Comprehensive report from model comparison.
MonteCarloStrategy<T> Monte Carlo cross-validation (repeated random sub-sampling validation).
NPVMetric<T> Computes Negative Predictive Value (NPV): proportion of negative predictions that are correct.
NegativeLikelihoodRatioMetric<T> Computes Negative Likelihood Ratio (LR-): (1 - Sensitivity) / Specificity = FNR / TNR.
NemenyiPostHocTest<T> Nemenyi post-hoc test: pairwise comparisons after Friedman test.
NemenyiResult<T> Results from Nemenyi post-hoc test.
NestedCVStrategy<T> Nested Cross-Validation: uses an inner CV loop for hyperparameter tuning and outer loop for evaluation.
NormalizedMSEMetric<T> Computes Normalized Mean Squared Error (NMSE): MSE divided by variance of actuals.
OptimizedPrecisionMetric<T> Computes Optimized Precision: Accuracy - |Sensitivity - Specificity| / (Sensitivity + Specificity).
PairedTTest<T> Paired t-test: compares means of two related samples.
PearsonCorrelationMetric<T> Computes Pearson Correlation Coefficient between predictions and actuals.
PoissonDevianceMetric<T> Computes Poisson Deviance for count data regression.
PositiveLikelihoodRatioMetric<T> Computes Positive Likelihood Ratio (LR+): Sensitivity / (1 - Specificity) = TPR / FPR.
PositivePredictiveValueMetric<T> Computes Positive Predictive Value (PPV): same as Precision but named differently in medical contexts.
PrecisionMetric<T> Computes precision (positive predictive value): the proportion of positive predictions that are correct.
PrevalenceMetric<T> Computes Prevalence: fraction of actual positives in the dataset.
PrevalenceThresholdMetric<T> Computes Prevalence Threshold: the prevalence at which the test would have 50% PPV.
ProbabilityCalibrator<T> Calibrates model outputs to produce reliable probability estimates.
PurgedKFoldStrategy<T> Purged K-Fold: K-Fold with temporal purging to prevent data leakage in financial/time-dependent data.
QuantileLossMetric<T> Computes Quantile Loss (Pinball Loss) for quantile regression.
R2ScoreMetric<T> Computes R² (coefficient of determination): proportion of variance explained by the model.
RMSEMetric<T> Computes Root Mean Squared Error (RMSE): square root of MSE.
RMSLEMetric<T> Computes Root Mean Squared Logarithmic Error (RMSLE): measures ratio errors rather than absolute errors.
RecallMetric<T> Computes recall (sensitivity, true positive rate): the proportion of actual positives correctly identified.
RegressionResults<T> Regression-specific results container.
RelativeAbsoluteErrorMetric<T> Computes Relative Absolute Error (RAE): sum of absolute errors relative to baseline.
RelativeSquaredErrorMetric<T> Computes Relative Squared Error (RSE): sum of squared errors relative to baseline model.
RepeatedKFoldStrategy<T> Repeated K-Fold: runs K-Fold multiple times with different random shuffles.
ResidualStatistics<T> Statistics about regression residuals.
RobustnessEngine<T> Engine for analyzing model robustness to input perturbations and noise.
RobustnessResult<T> Results from robustness analysis.
SMAPEMetric<T> Computes Symmetric Mean Absolute Percentage Error (SMAPE): a bounded percentage error metric.
ShuffleSplitStrategy<T> Shuffle Split (Monte Carlo Cross-Validation): random train/test splits repeated multiple times.
SlidingWindowStrategy<T> Sliding Window cross-validation for time series with fixed-size training window.
SpearmanCorrelationMetric<T> Computes Spearman’s Rank Correlation Coefficient between predictions and actuals.
SpecificityMetric<T> Computes specificity (true negative rate): the proportion of actual negatives correctly identified.
StatisticalTestEngine<T> Engine for performing statistical tests on model comparison results.
StatisticalTestResult<T> Represents the result of a statistical test.
StratifiedGroupKFoldStrategy<T> Stratified Group K-Fold: combines stratification and group constraints.
StratifiedKFoldStrategy<T> Stratified K-Fold: K-Fold that preserves the percentage of samples for each class.
SymmetricMAPEMetric<T> Computes Symmetric Mean Absolute Percentage Error (sMAPE) for regression.
TheilUMetric<T> Computes Theil’s U Statistic: measures forecast accuracy relative to a naive no-change forecast.
ThreatScoreMetric<T> Computes Threat Score (Critical Success Index): TP / (TP + FN + FP).
TimeSeriesSplitStrategy<T> Time Series Split: expanding window cross-validation that respects temporal order.
TrueNegativeRateMetric<T> Computes True Negative Rate (TNR): same as Specificity, TN / (TN + FP).
TweedieLossMetric<T> Computes Tweedie Deviance Loss for regression with power parameter.
ValidationCurveEngine<T> Engine for generating validation curves: how model performance changes with hyperparameter values.
ValidationCurveResult<T> Results from validation curve analysis.
WAPEMetric<T> Computes Weighted Absolute Percentage Error (WAPE): total absolute error as percentage of total actuals.
WeightedMAPEMetric<T> Computes Weighted Mean Absolute Percentage Error (wMAPE): weighted by actual values.
WilcoxonSignedRankTest<T> Wilcoxon signed-rank test: non-parametric paired comparison test.
ZeroOneLossMetric<T> Computes Zero-One Loss: fraction of misclassifications (complement of accuracy).

Interfaces (13)

Type Summary
IClassificationMetric<T> Interface for classification metrics.
IClassifierComparisonTest<T> Interface for tests comparing multiple classifier predictions against ground truth.
ICrossValidationStrategy<T> Defines a cross-validation splitting strategy.
IMetric<T> Base interface for all evaluation metrics.
IMultipleComparisonTest<T> Interface for tests comparing multiple classifiers or groups.
IMultipleSampleTest<T> Interface for statistical tests comparing multiple groups.
IPairedTest<T> Interface for paired comparison tests (e.g., comparing same samples under different conditions).
IProbabilisticClassificationMetric<T> Interface for classification metrics that use probabilities.
IRankingMetric<T> Interface for ranking/recommendation metrics.
IRegressionMetric<T> Interface for regression metrics.
IStatisticalTest<T> Interface for general statistical tests.
ITimeSeriesMetric<T> Interface for time series forecasting metrics.
ITwoSampleTest<T> Interface for statistical tests comparing two groups or samples.

Enums (27)

Type Summary
AdversarialAttackMethod Methods for generating adversarial examples.
AveragingMethod Specifies the averaging method for multi-class/multi-label classification metrics.
BiasVarianceDiagnosis Specifies the diagnosed bias-variance condition of a model.
BinningStrategy Strategies for binning predictions in calibration analysis.
CenterType<T>
ConfidenceIntervalMethod Specifies the method for computing confidence intervals on evaluation metrics.
ContinuousBinningStrategy Strategy for binning continuous features.
CrossValidationStrategy Specifies the cross-validation strategy to use for model evaluation.
EffectSizeMeasure Effect size measures for comparing models.
FairnessConstraint Specifies the fairness constraint or metric for model evaluation.
MetricDirection Specifies whether higher or lower values are better for a metric.
MissingValueStrategy Strategies for handling missing values in robustness testing.
MultipleComparisonTest Tests for comparing multiple models simultaneously.
MultipleTestingCorrectionMethod Methods for correcting p-values when performing multiple comparisons.
NormalizationType Specifies the normalization method for confusion matrices and metrics.
OptimalValueSelectionMethod Method for selecting optimal hyperparameter value from validation curve.
PairwiseComparisonTest Tests for pairwise model comparison.
PostHocTest Post-hoc tests after significant omnibus test.
PredictionIntervalMethod Methods for computing prediction intervals.
ReportDetailLevel Detail level for evaluation reports.
ReportFormat Specifies the output format for evaluation reports.
ReportSection Sections that can be included in evaluation reports.
SliceSortOrder Sort order for slices in reports.
ThresholdSelectionMethod Specifies the method for selecting classification thresholds.
UncertaintyCalibrationMethod Methods for calibrating uncertainty estimates.
UncertaintyDecompositionMethod Methods for decomposing predictive uncertainty into components.
WarningSeverity Severity levels for evaluation warnings.

Structs (1)

Type Summary
CVFold<T> Represents a single cross-validation fold with train and validation data.

Options & Configuration (13)

Type Summary
CalibrationOptions Configuration options for probability calibration analysis.
CrossValidationOptions Configuration options for cross-validation.
EvaluationOptions<T> Configuration options for model evaluation.
FairnessOptions Configuration options for fairness evaluation.
InfluenceAnalysisOptions Configuration options for influence analysis in regression models.
LearningCurveOptions Configuration options for learning curve analysis.
ModelComparisonOptions Configuration options for comparing multiple models.
ReportOptions Configuration options for evaluation report generation.
ResidualAnalysisOptions Configuration options for residual analysis in regression models.
RobustnessOptions Configuration options for robustness evaluation.
SubgroupAnalysisOptions Configuration options for subgroup (slice-based) analysis.
UncertaintyOptions Configuration options for uncertainty quantification.
ValidationCurveOptions Configuration options for validation curve analysis.