TfidfVectorizer<T>
Models & Types · AiDotNet.Preprocessing.TextVectorizers
Converts a collection of text documents to a TF-IDF weighted matrix.
Inheritance: TextVectorizerBase<T> → TfidfVectorizer<T>
For Beginners
TF-IDF makes rare but meaningful words more important:
- Common words like “the” appear everywhere, so they get low weight
- Rare words that only appear in specific documents get high weight
- This helps distinguish documents by their unique content
Example: “machine learning” in a tech article is more meaningful than “the” which appears in every document.
How It Works
TfidfVectorizer converts text to TF-IDF (Term Frequency-Inverse Document Frequency) representation, which weights terms by their importance in a document relative to the entire corpus.
TF-IDF = TF × IDF where:
- TF (Term Frequency) = count of term in document / total terms in document
- IDF (Inverse Document Frequency) = log(total documents / documents containing term)
Constructors
| Constructor | Summary |
|---|---|
TfidfVectorizer(int minDf, double maxDf, int? maxFeatures, ValueTuple<int, int>? nGramRange, bool lowercase, TfidfNorm norm, bool useIdf, bool smoothIdf, bool sublinearTf, Func<string, IEnumerable<string>> tokenizer, HashSet<string> stopWords, ITokenizer advancedTokenizer) |
Creates a new instance of TfidfVectorizer. |
Properties
| Property | Summary |
|---|---|
bool IsFitted { get; } |
|
double[] IdfWeights { get; } |
Gets the IDF weights for each term. |
Methods
| Method | Summary |
|---|---|
Matrix<T> Transform(IEnumerable<string> documents) |
Transforms documents to TF-IDF weighted vectors. |
void Fit(IEnumerable<string> documents) |
Fits the vectorizer to the corpus. |