TextVectorizerBase<T>

Base Classes · AiDotNet.Preprocessing.TextVectorizers

Base class for text vectorizers providing common functionality.

Implements: ITextVectorizer<T>

Known derived types: BM25Vectorizer<T>, BinaryVectorizer<T>, CharNGramVectorizer<T>, CountVectorizer<T>, Doc2VecVectorizer<T>, FastTextVectorizer<T>, GloVeVectorizer<T>, HashingVectorizer<T>, LDAVectorizer<T>, LSAVectorizer<T>, NMFVectorizer<T>, PretrainedEmbeddingsVectorizer<T>, SIFVectorizer<T>, TfidfVectorizer<T>, Word2VecVectorizer<T>

For Beginners

This is the foundation that all text vectorizers build upon. It handles the common task of breaking text into tokens and managing which words the vectorizer knows about. Specific vectorizers (TF-IDF, Count, etc.) add their own logic for how to convert those tokens into numbers.

How It Works

This base class provides shared implementation for text vectorization including:

  • Tokenization (splitting text into words/tokens)
  • N-gram generation (creating word sequences)
  • Stop word filtering
  • Vocabulary management

Supports two tokenization approaches:

  • Simple Tokenizer (default): Uses word-level tokenization via a custom function

or the built-in whitespace/punctuation splitter. Best for traditional NLP tasks.

  • ITokenizer Integration: Uses advanced subword tokenization (BPE, WordPiece, SentencePiece)

from the AiDotNet.Tokenization module. Enables consistent tokenization between classical ML and neural network approaches.

Properties

Property Summary
Dictionary<string, int> Vocabulary { get; }
HashSet<string> EnglishStopWords { get; } Common English stop words.
bool IsFitted { get; }
int FeatureCount { get; }
string[] FeatureNames { get; }

Methods

Method Summary
Matrix<T> FitTransform(IEnumerable<string> documents)
Matrix<T> Transform(IEnumerable<string> documents)
string[] GetFeatureNamesOut()
void Fit(IEnumerable<string> documents)