RetrievalMetrics
Helpers & Utilities · AiDotNet.RetrievalAugmentedGeneration.Evaluation
Pure-math information-retrieval (IR) metrics over ranked retrieval results: nDCG@k, MRR, MAP, Hit-Rate@k, Precision@k and Recall@k. These compare a ranked list of retrieved document ids against a set of ids known to be relevant. They involve no language model or embedding model and are therefore fully deterministic.
For Beginners
after your retriever returns a ranked list of documents, these functions grade the ranking. Precision@k asks “of the top k, how many were relevant?”; Recall@k asks “of all relevant documents, how many did we find in the top k?”; MRR rewards putting the first relevant document early; MAP and nDCG reward putting all relevant documents high up, with nDCG additionally discounting lower ranks smoothly.
How It Works
Conventions: Precision@k divides by k (the requested cutoff). Recall@k, MAP, MRR and nDCG return 0 when there are no relevant documents (the quantity is otherwise undefined). nDCG uses binary relevance in the primary overloads and log2 rank discounting.
Methods
| Method | Summary |
|---|---|
double AveragePrecision(IReadOnlyList<TId> ranked, ISet<TId> relevant) |
Average precision (AP) for a single query: the mean of Precision@k taken at every rank that holds a relevant item, divided by the total number of relevant items. |
double DcgAtK(IReadOnlyList<TId> ranked, ISet<TId> relevant, int k) |
Discounted cumulative gain at rank k with binary relevance and log2 discounting. |
double HitRateAtK(IReadOnlyList<TId> ranked, ISet<TId> relevant, int k) |
Hit-rate at rank k: 1 if at least one relevant item appears in the top-k, otherwise 0. |
double MeanAveragePrecision(IEnumerable<IReadOnlyList<TId>> rankings, IEnumerable<ISet<TId>> relevantSets) |
Mean average precision (MAP) across a set of queries. |
double MeanReciprocalRank(IEnumerable<IReadOnlyList<TId>> rankings, IEnumerable<ISet<TId>> relevantSets) |
Mean reciprocal rank (MRR) across a set of queries. |
double NdcgAtK(IReadOnlyList<TId> ranked, IReadOnlyDictionary<TId, double> gains, int k) |
Normalized discounted cumulative gain at rank k with graded (non-binary) relevance gains. |
double NdcgAtK(IReadOnlyList<TId> ranked, ISet<TId> relevant, int k) |
Normalized discounted cumulative gain at rank k with graded (non-binary) relevance gains. |
double PrecisionAtK(IReadOnlyList<TId> ranked, ISet<TId> relevant, int k) |
Precision at rank k: the fraction of the top-k retrieved items that are relevant. |
double RecallAtK(IReadOnlyList<TId> ranked, ISet<TId> relevant, int k) |
Recall at rank k: the fraction of all relevant items that appear in the top-k. |
double ReciprocalRank(IReadOnlyList<TId> ranked, ISet<TId> relevant) |
Reciprocal rank for a single query: 1 / (rank of the first relevant item), or 0 if none. |