TextVectorizerBase<T>
Base Classes · AiDotNet.Preprocessing.TextVectorizers
Base class for text vectorizers providing common functionality.
Implements: ITextVectorizer<T>
Known derived types: BM25Vectorizer<T>, BinaryVectorizer<T>, CharNGramVectorizer<T>, CountVectorizer<T>, Doc2VecVectorizer<T>, FastTextVectorizer<T>, GloVeVectorizer<T>, HashingVectorizer<T>, LDAVectorizer<T>, LSAVectorizer<T>, NMFVectorizer<T>, PretrainedEmbeddingsVectorizer<T>, SIFVectorizer<T>, TfidfVectorizer<T>, Word2VecVectorizer<T>
For Beginners
This is the foundation that all text vectorizers build upon. It handles the common task of breaking text into tokens and managing which words the vectorizer knows about. Specific vectorizers (TF-IDF, Count, etc.) add their own logic for how to convert those tokens into numbers.
How It Works
This base class provides shared implementation for text vectorization including:
- Tokenization (splitting text into words/tokens)
- N-gram generation (creating word sequences)
- Stop word filtering
- Vocabulary management
Supports two tokenization approaches:
- Simple Tokenizer (default): Uses word-level tokenization via a custom function
or the built-in whitespace/punctuation splitter. Best for traditional NLP tasks.
- ITokenizer Integration: Uses advanced subword tokenization (BPE, WordPiece, SentencePiece)
from the AiDotNet.Tokenization module. Enables consistent tokenization between classical ML and neural network approaches.
Properties
| Property | Summary |
|---|---|
Dictionary<string, int> Vocabulary { get; } |
|
HashSet<string> EnglishStopWords { get; } |
Common English stop words. |
bool IsFitted { get; } |
|
int FeatureCount { get; } |
|
string[] FeatureNames { get; } |
Methods
| Method | Summary |
|---|---|
Matrix<T> FitTransform(IEnumerable<string> documents) |
|
Matrix<T> Transform(IEnumerable<string> documents) |
|
string[] GetFeatureNamesOut() |
|
void Fit(IEnumerable<string> documents) |