StatisticsHelper<T>

Helpers & Utilities · AiDotNet.Helpers

Provides statistical calculation methods for various data analysis tasks.

For Beginners

This class contains methods to calculate common statistical measures like averages, variations, and statistical tests. These help you understand your data’s patterns and make decisions based on statistical evidence.

Methods

Method Summary
ChiSquareTestResult<T> ChiSquareTest(Vector<T> leftY, Vector<T> rightY, T significanceLevel) Performs a Chi-Square test to determine if there is a significant association between two categorical variables.
ConfusionMatrix<T> CalculateConfusionMatrix(Vector<T> actual, Vector<T> predicted, T threshold) Calculates a confusion matrix for binary classification at a specified threshold.
DistributionFitResult<T> DetermineBestFitDistribution(Vector<T> values) Analyzes a dataset and determines which statistical distribution best fits the data.
FTestResult<T> FTest(Vector<T> leftY, Vector<T> rightY, T significanceLevel) Performs an F-test to compare the variances of two samples.
IEnumerable<T> GeneratePosteriorPredictiveSamples(Matrix<T> features, Vector<T> coefficients, int numSamples) Generates samples from the posterior predictive distribution.
List<T> CalculateLearningCurve(Vector<T> yActual, Vector<T> yPredicted, int steps) Calculates a learning curve by evaluating model performance on increasingly larger subsets of data.
List<T> CalculateLeaveOneOutPredictiveDensities(Matrix<T> features, Vector<T> actualValues, Func<Matrix<T>, Vector<T>, Vector<T>> modelFitFunction) Calculates leave-one-out predictive densities for each observation.
List<T> CalculatePosteriorPredictiveSamples(Vector<T> actual, Vector<T> predicted, int featureCount, int numSamples) Generates samples from the posterior predictive distribution for a model.
List<T> CalculateVIF(Matrix<T> correlationMatrix, ModelStatsOptions options) Calculates the Variance Inflation Factor (VIF) for each feature based on a correlation matrix.
List<ValueTuple<T, T, T>> CalculateQuantileIntervals(Vector<T> actual, Vector<T> predicted, T[] quantiles) Calculates confidence intervals around specified quantiles of the predicted values.
MannWhitneyUTestResult<T> MannWhitneyUTest(Vector<T> leftY, Vector<T> rightY, T significanceLevel) Performs a Mann-Whitney U test to compare distributions between two groups.
Matrix<T> CalculateCorrelationMatrix(Matrix<T> features, ModelStatsOptions options) Calculates a correlation matrix for a set of features.
Matrix<T> CalculateCovarianceMatrix(Matrix<T> matrix) Calculates the covariance matrix for a dataset.
PermutationTestResult<T> PermutationTest(Vector<T> leftY, Vector<T> rightY, T significanceLevel) Performs a permutation test to determine if there is a significant difference between two groups.
T CalculateAIC(int sampleSize, int parameterSize, T rss) Calculates the Akaike Information Criterion (AIC) for model comparison.
T CalculateAICAlternative(int sampleSize, int parameterSize, T rss) Calculates an alternative formulation of the Akaike Information Criterion (AIC).
T CalculateAUC(Vector<T> fpr, Vector<T> tpr) Calculates the Area Under a Curve (AUC) given x and y coordinates.
T CalculateAccuracy(Vector<T> actual, Vector<T> predicted) Calculates the accuracy of predictions by comparing them to actual values.
T CalculateAccuracy(Vector<T> actual, Vector<T> predicted, PredictionType predictionType, T tolerance) Calculates the accuracy of predictions with support for different prediction types and tolerance levels.
T CalculateAccuracy(Vector<T> actual, Vector<T> predicted, PredictionType predictionType, T tolerance, bool treatDefaultAsMissing)
T CalculateAdjustedR2(T r2, int n, int p) Calculates the adjusted R² value, which accounts for the number of predictors in the model.
T CalculateAdjustedRandIndex(Vector<T> labels1, Vector<T> labels2) Calculates the Adjusted Rand Index (ARI) between two clusterings.
T CalculateBIC(int sampleSize, int parameterSize, T rss) Calculates the Bayesian Information Criterion (BIC) for model comparison.
T CalculateBayesFactor(ModelStats<T, TInput, TOutput> modelStats) Calculates the Bayes Factor for comparing two models.
T CalculateBetaCDF(T x, T alpha, T beta) Calculates the cumulative distribution function (CDF) of the Beta distribution.
T CalculateCRPS(Vector<T> actual, Vector<T> predicted) Calculates the CRPS for point predictions (assumes zero uncertainty).
T CalculateCRPS(Vector<T> actual, Vector<T> predictedMean, Vector<T> predictedStdDev) Calculates the Continuous Ranked Probability Score (CRPS) for probabilistic forecasts.
T CalculateCalinskiHarabaszIndex(Matrix<T> data, Vector<T> labels) Calculates the Calinski-Harabasz Index (Variance Ratio Criterion) for a clustering result.
T CalculateChiSquareCDF(int degreesOfFreedom, T x) Calculates the chi-square cumulative distribution function (CDF) value for a given chi-square value and degrees of freedom.
T CalculateChiSquarePDF(int degreesOfFreedom, T x) Calculates the probability density function (PDF) of the chi-square distribution.
T CalculateClopperPearsonLowerBound(int successes, int trials, T confidence) Calculates the Clopper-Pearson lower confidence bound for a binomial proportion.
T CalculateConditionNumber(Matrix<T> matrix, ModelStatsOptions options) Calculates the condition number of a matrix using the specified method.
T CalculateDIC(ModelStats<T, TInput, TOutput> modelStats) Calculates the Deviance Information Criterion (DIC) for Bayesian model comparison.
T CalculateDaviesBouldinIndex(Matrix<T> data, Vector<T> labels) Calculates the Davies-Bouldin Index for a clustering result.
T CalculateDistance(Vector<T> v1, Vector<T> v2, DistanceMetricType metric, Matrix<T> covarianceMatrix) Calculates the distance or similarity between two vectors using the specified metric.
T CalculateDurbinWatsonStatistic(List<T> residualList) Calculates the Durbin-Watson statistic from a list of residuals.
T CalculateDurbinWatsonStatistic(Vector<T> actual, Vector<T> predicted) Calculates the Durbin-Watson statistic to test for autocorrelation in residuals.
T CalculateDynamicTimeWarping(Vector<T> series1, Vector<T> series2) Calculates the Dynamic Time Warping (DTW) distance between two time series.
T CalculateEffectiveNumberOfParameters(Matrix<T> features, Vector<T> coefficients) Calculates the effective number of parameters in a model using the trace of the hat matrix.
T CalculateExplainedVarianceScore(Vector<T> actual, Vector<T> predicted) Calculates the explained variance score between actual and predicted values.
T CalculateExponentialPDF(T lambda, T x) Calculates the probability density function (PDF) value for an exponential distribution.
T CalculateF1Score(T precision, T recall) Calculates the F1 score from precision and recall values.
T CalculateFriedmanMSE(Vector<T> y, List<int> leftIndices, List<int> rightIndices) Calculates the Friedman Mean Squared Error for a potential split in a decision tree.
T CalculateGoodnessOfFit(Vector<T> values, Func<T, T> pdfFunction) Calculates the goodness of fit for a probability distribution against sample data.
T CalculateInverseBetaCDF(T probability, T alpha, T beta) Calculates the inverse cumulative distribution function (quantile function) of the Beta distribution.
T CalculateInverseChiSquareCDF(int degreesOfFreedom, T probability) Calculates the inverse of the chi-square cumulative distribution function (CDF).
T CalculateInverseExponentialCDF(T lambda, T probability) Calculates the inverse cumulative distribution function (CDF) of the exponential distribution.
T CalculateInverseLaplaceCDF(T median, T mad, T probability) Calculates the inverse of the Laplace cumulative distribution function (CDF).
T CalculateInverseNormalCDF(T mean, T stdDev, T probability) Calculates the inverse of the normal cumulative distribution function (CDF) with specified mean and standard deviation.
T CalculateInverseNormalCDF(T probability) Calculates the inverse of the standard normal cumulative distribution function (CDF).
T CalculateInverseStudentTCDF(int degreesOfFreedom, T probability) Calculates the inverse of the Student’s t cumulative distribution function (CDF).
T CalculateKendallTau(Vector<T> x, Vector<T> y) Calculates Kendall’s tau correlation coefficient between two vectors.
T CalculateLOO(ModelStats<T, TInput, TOutput> modelStats) Calculates the Leave-One-Out Cross-Validation (LOO-CV) criterion for Bayesian model comparison.
T CalculateLaplacePDF(T median, T mad, T x) Calculates the probability density function (PDF) value for a Laplace distribution.
T CalculateLikelihood(T actual, T predicted) Calculates the likelihood of an observed value given a predicted value.
T CalculateLogLikelihood(Vector<T> actualValues, Vector<T> predictedValues) Calculates the log-likelihood of a model given actual and predicted values.
T CalculateLogNormalPDF(T mu, T sigma, T x) Calculates the probability density function (PDF) value for a log-normal distribution.
T CalculateLogPointwisePredictiveDensity(Vector<T> actualValues, Vector<T> predictedValues) Calculates the log pointwise predictive density (LPPD) for a model.
T CalculateMarginalLikelihood(Vector<T> actualValues, Vector<T> predictedValues, int numParameters) Calculates an approximation of the marginal likelihood for a model.
T CalculateMaxError(Vector<T> actual, Vector<T> predicted) Calculates the maximum absolute error between actual and predicted values.
T CalculateMean(IEnumerable<T> values) Calculates the arithmetic mean (average) of a collection of values.
T CalculateMeanAbsoluteDeviation(Vector<T> values, T median) Calculates the Mean Absolute Deviation (MAD) of a vector of values from a given median.
T CalculateMeanAbsoluteError(Vector<T> actualValues, Vector<T> predictedValues) Calculates the Mean Absolute Error (MAE) between actual and predicted values.
T CalculateMeanAbsoluteError(Vector<T> y, List<int> leftIndices, List<int> rightIndices) Calculates the Mean Absolute Error (MAE) for a split in decision tree algorithms.
T CalculateMeanAbsolutePercentageError(Vector<T> actual, Vector<T> predicted) Calculates the Mean Absolute Percentage Error (MAPE) between actual and predicted values.
T CalculateMeanAveragePrecision(Vector<T> actual, Vector<T> predicted, int k) Calculates the Mean Average Precision (MAP) at k for a ranking task.
T CalculateMeanBiasError(Vector<T> actual, Vector<T> predicted) Calculates the mean bias error between actual and predicted values.
T CalculateMeanPredictionError(Vector<T> actual, Vector<T> predicted) Calculates the mean absolute prediction error between actual and predicted values.
T CalculateMeanReciprocalRank(Vector<T> actual, Vector<T> predicted) Calculates the Mean Reciprocal Rank (MRR) for a ranking task.
T CalculateMeanSquaredError(IEnumerable<T> actualValues, IEnumerable<T> predictedValues) Calculates the Mean Squared Error (MSE) between a set of values and their mean.
T CalculateMeanSquaredError(IEnumerable<T> values, T mean) Calculates the Mean Squared Error (MSE) between a set of values and their mean.
T CalculateMeanSquaredError(Vector<T> y, List<int> leftIndices, List<int> rightIndices) Calculates the Mean Squared Error (MSE) for a potential split in a decision tree.
T CalculateMeanSquaredLogError(Vector<T> actual, Vector<T> predicted) Calculates the Mean Squared Logarithmic Error (MSLE) between actual and predicted values.
T CalculateMedian(IEnumerable<T> values) Calculates the median value from a collection of numeric values.
T CalculateMedianAbsoluteDeviation(Vector<T> values) Calculates the median absolute deviation (MAD) of a set of values.
T CalculateMedianAbsoluteError(Vector<T> actual, Vector<T> predicted) Calculates the median absolute error between actual and predicted values.
T CalculateMedianPredictionError(Vector<T> actual, Vector<T> predicted) Calculates the median absolute prediction error between actual and predicted values.
T CalculateMutualInformation(Vector<T> x, Vector<T> y) Calculates the mutual information between two discrete random variables.
T CalculateNDCG(Vector<T> actual, Vector<T> predicted, int k) Calculates the Normalized Discounted Cumulative Gain (NDCG) at k for a ranking task.
T CalculateNormalCDF(T mean, T stdDev, T x) Calculates the Cumulative Distribution Function (CDF) for a normal distribution.
T CalculateNormalPDF(T mean, T stdDev, T x) Calculates the probability density function (PDF) for a normal (Gaussian) distribution.
T CalculateNormalizedMutualInformation(Vector<T> x, Vector<T> y) Calculates the normalized mutual information between two variables.
T CalculateObservedTestStatistic(Vector<T> actualValues, Vector<T> predictedValues, TestStatisticType testType) Calculates a test statistic comparing actual and predicted values.
T CalculatePValue(Vector<T> leftY, Vector<T> rightY, TestStatisticType testType) Calculates the p-value for a statistical test comparing two groups.
T CalculatePeakDifference(Vector<T> x1, Vector<T> y1, Vector<T> x2, Vector<T> y2) Calculates the difference between peak values in two distributions.
T CalculatePearsonCorrelation(Vector<T> x, Vector<T> y) Calculates the Pearson correlation coefficient between two sets of values.
T CalculatePearsonCorrelationCoefficient(Vector<T> x, Vector<T> y) Calculates the Pearson correlation coefficient between two vectors.
T CalculatePopulationStandardError(Vector<T> actual, Vector<T> predicted) Calculates the population standard error of the estimate.
T CalculatePosteriorPredictiveCheck(ModelStats<T, TInput, TOutput> modelStats) Calculates a posterior predictive p-value for model checking.
T CalculatePrecisionRecallAUC(Vector<T> actual, Vector<T> predicted) Calculates the area under the precision-recall curve (PR AUC).
T CalculatePredictionIntervalCoverage(Vector<T> actual, Vector<T> predicted, T lowerInterval, T upperInterval) Calculates the proportion of actual values that fall within a specified prediction interval.
T CalculateQuantile(T[] sortedData, T quantile) Calculates a specific quantile from sorted data.
T CalculateR2(Vector<T> actualValues, Vector<T> predictedValues) Calculates the coefficient of determination (R²) between actual and predicted values.
T CalculateROCAUC(Vector<T> actual, Vector<T> predicted) Calculates the Area Under the Receiver Operating Characteristic Curve (ROC AUC).
T CalculateReferenceModelMarginalLikelihood(Vector<T> actual) Calculates the marginal likelihood for a reference model (intercept-only model).
T CalculateResidualSumOfSquares(Vector<T> actual, Vector<T> predicted) Calculates the residual sum of squares (SSE) between actual and predicted values.
T CalculateRootMeanSquaredError(Vector<T> actualValues, Vector<T> predictedValues) Calculates the Root Mean Squared Error (RMSE) between actual and predicted values.
T CalculateSampleStandardError(Vector<T> actual, Vector<T> predicted, int numberOfParameters) Calculates the sample standard error of the estimate, adjusted for the number of model parameters.
T CalculateSilhouetteScore(Matrix<T> data, Vector<T> labels) Calculates the silhouette score for a clustering result.
T CalculateSpearmanRankCorrelationCoefficient(Vector<T> x, Vector<T> y) Calculates the Spearman rank correlation coefficient between two vectors.
T CalculateSplitScore(Vector<T> y, List<int> leftIndices, List<int> rightIndices, SplitCriterion splitCriterion) Calculates a score for a data split based on the specified criterion.
T CalculateStandardDeviation(IEnumerable<T> values) Calculates the standard deviation of a vector.
T CalculateStudentPDF(T x, T mean, T stdDev, int df) Calculates the probability density function (PDF) value for a Student’s t-distribution.
T CalculateSymmetricMeanAbsolutePercentageError(Vector<T> actual, Vector<T> predicted) Calculates the Symmetric Mean Absolute Percentage Error (SMAPE) between actual and predicted values.
T CalculateTValue(int degreesOfFreedom, T confidenceLevel) Calculates the t-value for a given degrees of freedom and confidence level.
T CalculateTheilUStatistic(Vector<T> actual, Vector<T> predicted) Calculates Theil’s U statistic, a measure of forecast accuracy.
T CalculateTotalSumOfSquares(Vector<T> values) Calculates the total sum of squares (SST) for a set of values.
T CalculateVariance(IEnumerable<T> values) Calculates the variance of a collection of values.
T CalculateVariance(Vector<T> values, T mean) Calculates the variance of a vector of values from a given mean.
T CalculateVarianceReduction(Vector<T> y, List<int> leftIndices, List<int> rightIndices) Calculates the variance reduction achieved by splitting data into left and right groups.
T CalculateVariationOfInformation(Vector<T> x, Vector<T> y) Calculates the variation of information (also known as shared information distance) between two variables.
T CalculateWAIC(ModelStats<T, TInput, TOutput> modelStats) Calculates the Widely Applicable Information Criterion (WAIC) for Bayesian model comparison.
T CalculateWeibullPDF(T k, T lambda, T x) Calculates the probability density function (PDF) value for a Weibull distribution.
T CosineSimilarity(Vector<T> v1, Vector<T> v2) Calculates the cosine similarity between two vectors.
T Digamma(T x) Calculates the digamma function, which is the logarithmic derivative of the gamma function.
T EuclideanDistance(Vector<T> v1, Vector<T> v2) Calculates the Euclidean distance between two vectors.
T Gamma(T x) Calculates the gamma function for a given value.
T HammingDistance(Vector<T> v1, Vector<T> v2) Calculates the Hamming distance between two vectors.
T IncompleteGamma(T a, T x) Calculates the incomplete gamma function, which is used in various statistical distributions.
T JaccardSimilarity(Vector<T> v1, Vector<T> v2) Calculates the Jaccard similarity coefficient between two vectors.
T MahalanobisDistance(Vector<T> v1, Vector<T> v2, Matrix<T> covarianceMatrix) Calculates the Mahalanobis distance between two vectors, accounting for correlations in the data.
T ManhattanDistance(Vector<T> v1, Vector<T> v2) Calculates the Manhattan distance (L1 norm) between two vectors.
TTestResult<T> TTest(Vector<T> leftY, Vector<T> rightY, T significanceLevel) Performs a Student’s t-test to compare means between two groups.
ValueTuple<T, T, T> CalculatePrecisionRecallF1(Vector<T> actual, Vector<T> predicted, PredictionType predictionType, T threshold) Calculates precision, recall, and F1 score for a set of predictions.
ValueTuple<T, T, T> CalculatePrecisionRecallF1(Vector<T> actual, Vector<T> predicted, PredictionType predictionType, T threshold, bool treatDefaultAsMissing)
ValueTuple<T, T> CalculateAucF1Score(ModelEvaluationData<T, TInput, TOutput> evaluationData) Calculates both the AUC and F1 score for model evaluation.
ValueTuple<T, T> CalculateBootstrapInterval(Vector<T> actual, Vector<T> predicted, T confidenceLevel) Calculates confidence intervals using bootstrap resampling.
ValueTuple<T, T> CalculateClopperPearsonInterval(int successes, int trials, T confidence) Calculates the Clopper-Pearson (exact) confidence interval for a binomial proportion.
ValueTuple<T, T> CalculateConfidenceIntervals(Vector<T> values, T confidenceLevel, DistributionType distributionType) Calculates confidence intervals for the mean of a set of values based on the specified distribution type and confidence level.
ValueTuple<T, T> CalculateCredibleIntervals(Vector<T> values, T confidenceLevel, DistributionType distributionType) Calculates the credible intervals for a given set of values based on the specified distribution type and confidence level.
ValueTuple<T, T> CalculateForecastInterval(Vector<T> actual, Vector<T> predicted, T confidenceLevel) Calculates a forecast interval for future predictions.
ValueTuple<T, T> CalculateJackknifeInterval(Vector<T> actual, Vector<T> predicted) Calculates confidence intervals using the jackknife resampling method.
ValueTuple<T, T> CalculateMeanAndStandardDeviation(Vector<T> values) Calculates the mean (average) and standard deviation of a set of values.
ValueTuple<T, T> CalculatePercentileInterval(Vector<T> predicted, T confidenceLevel) Calculates a percentile-based confidence interval from predicted values.
ValueTuple<T, T> CalculatePredictionIntervals(Vector<T> actual, Vector<T> predicted, T confidenceLevel) Calculates prediction intervals for future observations based on a model’s predictions.
ValueTuple<T, T> CalculateQuantiles(Vector<T> data) Calculates the first and third quartiles (25th and 75th percentiles) of a data set.
ValueTuple<T, T> CalculateSimultaneousPredictionInterval(Vector<T> actual, Vector<T> predicted, T confidenceLevel) Calculates a simultaneous prediction interval for multiple future observations.
ValueTuple<T, T> CalculateSkewnessAndKurtosis(Vector<T> sample, T mean, T stdDev, int n) Calculates the skewness and kurtosis of a sample.
ValueTuple<T, T> CalculateToleranceInterval(Vector<T> actual, Vector<T> predicted, T confidenceLevel) Calculates a tolerance interval for a set of predicted values.
ValueTuple<T, T> CalculateWeibullConfidenceIntervals(Vector<T> values, T confidenceLevel) Calculates confidence intervals for Weibull distribution parameters using bootstrap resampling.
ValueTuple<T, T> CalculateWeibullCredibleIntervals(Vector<T> sample, T lowerProbability, T upperProbability) Calculates the credible intervals for a Weibull distribution.
ValueTuple<T, T> EstimateWeibullParameters(Vector<T> values) Estimates the shape and scale parameters of a Weibull distribution from sample data.
ValueTuple<Vector<T>, Vector<T>> CalculateROCCurve(Vector<T> actualValues, Vector<T> predictedValues) Calculates the Receiver Operating Characteristic (ROC) curve for a set of predictions.
Vector<T> CalculateAutoCorrelationFunction(Vector<T> series, int maxLag) Calculates the autocorrelation function (ACF) for a time series up to a specified maximum lag.
Vector<T> CalculatePartialAutoCorrelationFunction(Vector<T> series, int maxLag) Calculates the partial autocorrelation function (PACF) for a time series up to a specified maximum lag.
Vector<T> CalculateResiduals(Vector<T> actual, Vector<T> predicted) Calculates the residuals (errors) between actual and predicted values.
Vector<T> GenerateThresholds(Vector<T> predictedValues) Generates a set of unique threshold values from predicted values.