FastTextVectorizer<T>

Models & Types · AiDotNet.Preprocessing.TextVectorizers

Converts text documents to dense vectors using FastText-style subword embeddings.

Inheritance: TextVectorizerBase<T> → FastTextVectorizer<T>

For Beginners

FastText is Word2Vec that understands word parts:

  • Can generate vectors for words it has never seen before
  • “unhappiness” can be understood from “un-”, “happy”, “-ness” patterns
  • Better for languages with rich morphology (German, Turkish, Finnish)
  • Handles typos and spelling variations better than Word2Vec

How It Works

FastText extends Word2Vec by representing each word as a bag of character n-grams. This allows the model to:

  • Handle out-of-vocabulary (OOV) words by composing subword vectors
  • Capture morphological information (prefixes, suffixes, roots)
  • Be more robust to spelling variations and typos

For example, “where” with n-grams (3,6) includes: “<wh”, “whe”, “her”, “ere”, “re>”, “<whe”, “wher”, “here”, “ere>”, “<wher”, “where”, “here>”, “<where”, “where>”, “”

Constructors

Constructor Summary
FastTextVectorizer(int vectorSize, int windowSize, int minCount, int epochs, double learningRate, int negativeSamples, ValueTuple<int, int>? subwordNGramRange, int buckets, Word2VecAggregation aggregation, bool lowercase, int? randomState, Func<string, IEnumerable<string>> tokenizer, HashSet<string> stopWords, ITokenizer advancedTokenizer) Creates a new instance of FastTextVectorizer.

Properties

Property Summary
Dictionary<string, double[]> WordVectors { get; } Gets the learned word vectors for known words.
bool IsFitted { get; }
int FeatureCount { get; }
int VectorSize { get; } Gets the vector dimensionality.

Methods

Method Summary
Matrix<T> Transform(IEnumerable<string> documents) Transforms documents to dense vectors by aggregating word vectors.
double[] GetWordVector(string word) Gets the vector for a word, computing from subwords if unknown.
void Fit(IEnumerable<string> documents) Trains FastText embeddings on the corpus.