ITextVectorizer<T>

Interfaces · AiDotNet.Interfaces

Defines a text vectorizer that converts text documents to numeric feature matrices.

Known implementations: TextVectorizerBase<T>

For Beginners

A text vectorizer converts words into numbers that ML models can understand. Different vectorizers use different strategies:

  • TF-IDF: Weights words by importance (rare words score higher)
  • Count: Simply counts how many times each word appears
  • Hashing: Uses hashing for memory efficiency with large vocabularies
  • BM25: Improved TF-IDF used by search engines
  • Word2Vec/Doc2Vec: Creates dense embeddings capturing semantic meaning

How It Works

Text vectorizers transform collections of text documents into numeric representations suitable for machine learning algorithms. They follow the sklearn-style Fit/Transform pattern where the vectorizer first learns from training data, then transforms any text.

Properties

Property Summary
Dictionary<string, int> Vocabulary { get; } Gets the vocabulary mapping (token to index).
bool IsFitted { get; } Gets whether this vectorizer has been fitted to data.
int FeatureCount { get; } Gets the number of features (vocabulary size) this vectorizer produces.
string[] FeatureNames { get; } Gets the feature names (vocabulary terms in order).

Methods

Method Summary
Matrix<T> FitTransform(IEnumerable<string> documents) Fits the vectorizer and transforms the documents in one step.
Matrix<T> Transform(IEnumerable<string> documents) Transforms documents to a numeric feature matrix.
string[] GetFeatureNamesOut() Gets the output feature names.
void Fit(IEnumerable<string> documents) Fits the vectorizer to the training documents, learning the vocabulary.