BM25Vectorizer<T>

Models & Types · AiDotNet.Preprocessing.TextVectorizers

Converts text documents to BM25-weighted feature vectors.

Inheritance: TextVectorizerBase<T> → BM25Vectorizer<T>

For Beginners

BM25 is like TF-IDF but smarter:

  • Long documents don’t unfairly dominate (length normalization)
  • Repeating a word 100 times doesn’t score 100x better than once (saturation)
  • Used by Google, Elasticsearch, and most modern search engines
  • Generally performs better than TF-IDF for search and retrieval tasks

How It Works

BM25 (Best Matching 25) is an advanced ranking function used by search engines like Elasticsearch and Lucene. It improves upon TF-IDF by adding document length normalization and term frequency saturation.

The BM25 formula is:

Where:

  • f(qi,D) = term frequency of qi in document D
  • |D| = document length
  • avgdl = average document length in the corpus
  • k1 = term frequency saturation parameter (typically 1.2-2.0)
  • b = document length normalization parameter (typically 0.75)

Constructors

Constructor Summary
BM25Vectorizer(double k1, double b, double delta, int minDf, double maxDf, int? maxFeatures, ValueTuple<int, int>? nGramRange, bool lowercase, BM25Norm norm, Func<string, IEnumerable<string>> tokenizer, HashSet<string> stopWords, ITokenizer advancedTokenizer) Creates a new instance of BM25Vectorizer.

Properties

Property Summary
bool IsFitted { get; }
double AverageDocumentLength { get; } Gets the average document length in the corpus.
double[] IdfWeights { get; } Gets the IDF weights for each term.

Methods

Method Summary
Matrix<T> Transform(IEnumerable<string> documents) Transforms documents to BM25-weighted vectors.
void Fit(IEnumerable<string> documents) Fits the vectorizer to the corpus.