TextRecognitionMetrics
Helpers & Utilities · AiDotNet.Metrics
Evaluation metrics for text recognition (OCR): Character Error Rate, Word Error Rate, normalized edit distance and exact-match accuracy.
For Beginners
Error rates answer “what fraction of the text did the model get wrong?” A CER of 0.05 means about 5 characters in every 100 needed fixing. Note that an error rate can exceed 1.0: if the model emits far more text than the reference contains, the number of edits can be larger than the reference length. Accuracy metrics run the other way - higher is better - so always check which direction a reported number points.
How It Works
Recognition output is a string, so quality is measured by how many edits turn the prediction into the reference. All the metrics here are built on Levenshtein distance - the smallest number of single-character insertions, deletions and substitutions that transform one string into another.
Corpus versus sentence averaging. The overloads taking a whole list are corpus-level: they sum the edit distances and divide by the summed reference length. That is the convention used by the ICDAR Robust Reading competitions and by speech recognition. Averaging the per-sample rates instead would let a single short reference dominate, so prefer the corpus overloads when reporting a benchmark number.
Which number to report:
the standard number for line- and page-level OCR. Lower is better; 0 is perfect.
the same idea over whitespace-separated tokens, for document OCR.
ICDAR 1-NED. Higher is better; 1 is perfect. This is the one scene-text papers quote.
word-level accuracy for cropped-word benchmarks (IIIT5K, SVT, IC13, IC15). The field convention is case-insensitive and alphanumeric-only, which is this method default.
Methods
| Method | Summary |
|---|---|
double CharacterErrorRate(IReadOnlyList<string> references, IReadOnlyList<string> hypotheses) |
Computes the corpus-level Character Error Rate: the summed edit distance over the summed reference length. |
double CharacterErrorRate(string reference, string hypothesis) |
Computes the corpus-level Character Error Rate: the summed edit distance over the summed reference length. |
double ExactMatchAccuracy(IReadOnlyList<string> references, IReadOnlyList<string> hypotheses, bool caseSensitive, bool alphanumericOnly) |
Computes exact-match accuracy: the fraction of predictions that equal their reference after normalization. |
double NormalizedEditDistance(IReadOnlyList<string> references, IReadOnlyList<string> hypotheses) |
Computes mean ICDAR 1-NED over a dataset: the average of the per-sample NormalizedEditDistance. |
double NormalizedEditDistance(string reference, string hypothesis) |
Computes mean ICDAR 1-NED over a dataset: the average of the per-sample NormalizedEditDistance. |
double WordErrorRate(IReadOnlyList<string> references, IReadOnlyList<string> hypotheses) |
Computes the corpus-level Word Error Rate: the summed token edit distance over the summed reference token count. |
double WordErrorRate(string reference, string hypothesis) |
Computes the corpus-level Word Error Rate: the summed token edit distance over the summed reference token count. |
int LevenshteinDistance(string source, string target) |
Computes the Levenshtein edit distance between two strings: the minimum number of single-character insertions, deletions and substitutions needed to turn source into target. |
int TokenEditDistance(IReadOnlyList<string> source, IReadOnlyList<string> target) |
Computes the Levenshtein edit distance between two token sequences, used by WordErrorRate. |
string Normalize(string value, bool caseSensitive, bool alphanumericOnly) |
Applies the benchmark normalization used by ExactMatchAccuracy. |