TextRecognitionMetrics

Helpers & Utilities · AiDotNet.Metrics

Evaluation metrics for text recognition (OCR): Character Error Rate, Word Error Rate, normalized edit distance and exact-match accuracy.

For Beginners

Error rates answer “what fraction of the text did the model get wrong?” A CER of 0.05 means about 5 characters in every 100 needed fixing. Note that an error rate can exceed 1.0: if the model emits far more text than the reference contains, the number of edits can be larger than the reference length. Accuracy metrics run the other way - higher is better - so always check which direction a reported number points.

How It Works

Recognition output is a string, so quality is measured by how many edits turn the prediction into the reference. All the metrics here are built on Levenshtein distance - the smallest number of single-character insertions, deletions and substitutions that transform one string into another.

Corpus versus sentence averaging. The overloads taking a whole list are corpus-level: they sum the edit distances and divide by the summed reference length. That is the convention used by the ICDAR Robust Reading competitions and by speech recognition. Averaging the per-sample rates instead would let a single short reference dominate, so prefer the corpus overloads when reporting a benchmark number.

Which number to report:

the standard number for line- and page-level OCR. Lower is better; 0 is perfect.

the same idea over whitespace-separated tokens, for document OCR.

ICDAR 1-NED. Higher is better; 1 is perfect. This is the one scene-text papers quote.

word-level accuracy for cropped-word benchmarks (IIIT5K, SVT, IC13, IC15). The field convention is case-insensitive and alphanumeric-only, which is this method default.

Methods

Method Summary
double CharacterErrorRate(IReadOnlyList<string> references, IReadOnlyList<string> hypotheses) Computes the corpus-level Character Error Rate: the summed edit distance over the summed reference length.
double CharacterErrorRate(string reference, string hypothesis) Computes the corpus-level Character Error Rate: the summed edit distance over the summed reference length.
double ExactMatchAccuracy(IReadOnlyList<string> references, IReadOnlyList<string> hypotheses, bool caseSensitive, bool alphanumericOnly) Computes exact-match accuracy: the fraction of predictions that equal their reference after normalization.
double NormalizedEditDistance(IReadOnlyList<string> references, IReadOnlyList<string> hypotheses) Computes mean ICDAR 1-NED over a dataset: the average of the per-sample NormalizedEditDistance.
double NormalizedEditDistance(string reference, string hypothesis) Computes mean ICDAR 1-NED over a dataset: the average of the per-sample NormalizedEditDistance.
double WordErrorRate(IReadOnlyList<string> references, IReadOnlyList<string> hypotheses) Computes the corpus-level Word Error Rate: the summed token edit distance over the summed reference token count.
double WordErrorRate(string reference, string hypothesis) Computes the corpus-level Word Error Rate: the summed token edit distance over the summed reference token count.
int LevenshteinDistance(string source, string target) Computes the Levenshtein edit distance between two strings: the minimum number of single-character insertions, deletions and substitutions needed to turn source into target.
int TokenEditDistance(IReadOnlyList<string> source, IReadOnlyList<string> target) Computes the Levenshtein edit distance between two token sequences, used by WordErrorRate.
string Normalize(string value, bool caseSensitive, bool alphanumericOnly) Applies the benchmark normalization used by ExactMatchAccuracy.