DiversityReranker<T>
Models & Types · AiDotNet.RetrievalAugmentedGeneration.Rerankers
Reranks documents to maximize diversity while maintaining relevance.
Inheritance: RerankerBase<T> → DiversityReranker<T>
For Beginners
This prevents showing the same information multiple times.
The Problem: Imagine searching for “Python programming” and getting 10 results:
- Result 1: “Python is a programming language…”
- Result 2: “Python is a programming language used for…”
- Result 3: “Python programming language allows…”
- Result 4-10: More variations of the same thing
That’s redundant! You want variety:
- Result 1: Python basics
- Result 2: Python web development
- Result 3: Python data science
- Result 4: Python machine learning
- Result 5: Python performance tips
How it works:
- Pick the most relevant document first
- For remaining docs, balance two factors:
a) Relevance to the query (should be useful) b) Difference from already-picked docs (should be unique)
- Keep picking until you have enough results
Diversity calculation:
- Compares text overlap (how many words are shared)
- Higher overlap = less diverse = lower score
- Lower overlap = more diverse = higher score
Lambda parameter (0 to 1):
- lambda=1.0: Only care about relevance (might get duplicates)
- lambda=0.0: Only care about diversity (might get irrelevant docs)
- lambda=0.5: Balance both (recommended default)
Real example with lambda=0.5: Query: “climate change effects”
Step 1: Pick most relevant → “Climate change causes rising temperatures” (relevance: 0.9) Step 2: Next candidates:
- “Climate change leads to warmer weather” (relevance: 0.85, similarity to picked: 0.7)
→ Score: 0.5 * 0.85 - 0.5 * 0.7 = 0.075
- “Ocean acidification from CO2” (relevance: 0.7, similarity: 0.2)
→ Score: 0.5 * 0.7 - 0.5 * 0.2 = 0.25 ✓ Pick this!
Result: You get coverage of temperature AND ocean effects, not just temperature twice!
When to use this:
- Search results where redundancy is common
- Document recommendation systems
- Exploratory searches where breadth matters
- After initial retrieval that returns many similar docs
How It Works
This reranker addresses the problem of redundant results by explicitly promoting diversity. It uses a greedy algorithm to select documents that are both relevant to the query and dissimilar from already-selected documents. This is similar to Maximal Marginal Relevance (MMR) but uses a simpler diversity metric based on text overlap.
Constructors
| Constructor | Summary |
|---|---|
DiversityReranker(T lambda) |
Initializes a new instance of the DiversityReranker class. |
Properties
| Property | Summary |
|---|---|
bool ModifiesScores { get; } |
Gets a value indicating whether this reranker modifies relevance scores. |