BM25Vectorizer<T>
Models & Types · AiDotNet.Preprocessing.TextVectorizers
Converts text documents to BM25-weighted feature vectors.
Inheritance: TextVectorizerBase<T> → BM25Vectorizer<T>
For Beginners
BM25 is like TF-IDF but smarter:
- Long documents don’t unfairly dominate (length normalization)
- Repeating a word 100 times doesn’t score 100x better than once (saturation)
- Used by Google, Elasticsearch, and most modern search engines
- Generally performs better than TF-IDF for search and retrieval tasks
How It Works
BM25 (Best Matching 25) is an advanced ranking function used by search engines like Elasticsearch and Lucene. It improves upon TF-IDF by adding document length normalization and term frequency saturation.
The BM25 formula is:
Where:
- f(qi,D) = term frequency of qi in document D
- |D| = document length
- avgdl = average document length in the corpus
- k1 = term frequency saturation parameter (typically 1.2-2.0)
- b = document length normalization parameter (typically 0.75)
Constructors
| Constructor | Summary |
|---|---|
BM25Vectorizer(double k1, double b, double delta, int minDf, double maxDf, int? maxFeatures, ValueTuple<int, int>? nGramRange, bool lowercase, BM25Norm norm, Func<string, IEnumerable<string>> tokenizer, HashSet<string> stopWords, ITokenizer advancedTokenizer) |
Creates a new instance of BM25Vectorizer. |
Properties
| Property | Summary |
|---|---|
bool IsFitted { get; } |
|
double AverageDocumentLength { get; } |
Gets the average document length in the corpus. |
double[] IdfWeights { get; } |
Gets the IDF weights for each term. |
Methods
| Method | Summary |
|---|---|
Matrix<T> Transform(IEnumerable<string> documents) |
Transforms documents to BM25-weighted vectors. |
void Fit(IEnumerable<string> documents) |
Fits the vectorizer to the corpus. |