TfidfVectorizer<T>

Models & Types · AiDotNet.Preprocessing.TextVectorizers

Converts a collection of text documents to a TF-IDF weighted matrix.

Inheritance: TextVectorizerBase<T> → TfidfVectorizer<T>

For Beginners

TF-IDF makes rare but meaningful words more important:

  • Common words like “the” appear everywhere, so they get low weight
  • Rare words that only appear in specific documents get high weight
  • This helps distinguish documents by their unique content

Example: “machine learning” in a tech article is more meaningful than “the” which appears in every document.

How It Works

TfidfVectorizer converts text to TF-IDF (Term Frequency-Inverse Document Frequency) representation, which weights terms by their importance in a document relative to the entire corpus.

TF-IDF = TF × IDF where:

  • TF (Term Frequency) = count of term in document / total terms in document
  • IDF (Inverse Document Frequency) = log(total documents / documents containing term)

Constructors

Constructor Summary
TfidfVectorizer(int minDf, double maxDf, int? maxFeatures, ValueTuple<int, int>? nGramRange, bool lowercase, TfidfNorm norm, bool useIdf, bool smoothIdf, bool sublinearTf, Func<string, IEnumerable<string>> tokenizer, HashSet<string> stopWords, ITokenizer advancedTokenizer) Creates a new instance of TfidfVectorizer.

Properties

Property Summary
bool IsFitted { get; }
double[] IdfWeights { get; } Gets the IDF weights for each term.

Methods

Method Summary
Matrix<T> Transform(IEnumerable<string> documents) Transforms documents to TF-IDF weighted vectors.
void Fit(IEnumerable<string> documents) Fits the vectorizer to the corpus.