RetrievalMetrics

Helpers & Utilities · AiDotNet.RetrievalAugmentedGeneration.Evaluation

Pure-math information-retrieval (IR) metrics over ranked retrieval results: nDCG@k, MRR, MAP, Hit-Rate@k, Precision@k and Recall@k. These compare a ranked list of retrieved document ids against a set of ids known to be relevant. They involve no language model or embedding model and are therefore fully deterministic.

For Beginners

after your retriever returns a ranked list of documents, these functions grade the ranking. Precision@k asks “of the top k, how many were relevant?”; Recall@k asks “of all relevant documents, how many did we find in the top k?”; MRR rewards putting the first relevant document early; MAP and nDCG reward putting all relevant documents high up, with nDCG additionally discounting lower ranks smoothly.

How It Works

Conventions: Precision@k divides by k (the requested cutoff). Recall@k, MAP, MRR and nDCG return 0 when there are no relevant documents (the quantity is otherwise undefined). nDCG uses binary relevance in the primary overloads and log2 rank discounting.

Methods

Method Summary
double AveragePrecision(IReadOnlyList<TId> ranked, ISet<TId> relevant) Average precision (AP) for a single query: the mean of Precision@k taken at every rank that holds a relevant item, divided by the total number of relevant items.
double DcgAtK(IReadOnlyList<TId> ranked, ISet<TId> relevant, int k) Discounted cumulative gain at rank k with binary relevance and log2 discounting.
double HitRateAtK(IReadOnlyList<TId> ranked, ISet<TId> relevant, int k) Hit-rate at rank k: 1 if at least one relevant item appears in the top-k, otherwise 0.
double MeanAveragePrecision(IEnumerable<IReadOnlyList<TId>> rankings, IEnumerable<ISet<TId>> relevantSets) Mean average precision (MAP) across a set of queries.
double MeanReciprocalRank(IEnumerable<IReadOnlyList<TId>> rankings, IEnumerable<ISet<TId>> relevantSets) Mean reciprocal rank (MRR) across a set of queries.
double NdcgAtK(IReadOnlyList<TId> ranked, IReadOnlyDictionary<TId, double> gains, int k) Normalized discounted cumulative gain at rank k with graded (non-binary) relevance gains.
double NdcgAtK(IReadOnlyList<TId> ranked, ISet<TId> relevant, int k) Normalized discounted cumulative gain at rank k with graded (non-binary) relevance gains.
double PrecisionAtK(IReadOnlyList<TId> ranked, ISet<TId> relevant, int k) Precision at rank k: the fraction of the top-k retrieved items that are relevant.
double RecallAtK(IReadOnlyList<TId> ranked, ISet<TId> relevant, int k) Recall at rank k: the fraction of all relevant items that appear in the top-k.
double ReciprocalRank(IReadOnlyList<TId> ranked, ISet<TId> relevant) Reciprocal rank for a single query: 1 / (rank of the first relevant item), or 0 if none.