SIFVectorizer<T>

Models & Types · AiDotNet.Preprocessing.TextVectorizers

Converts text documents to sentence embeddings using Smooth Inverse Frequency (SIF).

Inheritance: TextVectorizerBase<T> → SIFVectorizer<T>

For Beginners

SIF creates sentence embeddings that are surprisingly good:

  • Simple: just weighted averaging of word vectors
  • Effective: often competitive with complex deep learning methods
  • Fast: no neural network inference required
  • Requires pre-trained word vectors (Word2Vec, GloVe, etc.)

How It Works

SIF is a simple but effective method for creating sentence embeddings from word vectors. It computes a weighted average of word vectors using smooth inverse frequency weights, then removes the first principal component to create better sentence representations.

The algorithm:

  1. Compute weighted average: v_s = (1/|s|) * Σ (a / (a + p(w))) * v_w
  2. Remove first principal component: v_s = v_s - u * u^T * v_s

Constructors

Constructor Summary
SIFVectorizer(Dictionary<string, double[]> wordVectors, double alpha, bool removePrincipalComponent, int nPrincipalComponents, bool lowercase, Func<string, IEnumerable<string>> tokenizer, HashSet<string> stopWords, ITokenizer advancedTokenizer) Creates a new instance of SIFVectorizer.

Properties

Property Summary
bool IsFitted { get; }
int FeatureCount { get; }
int VectorSize { get; } Gets the vector dimensionality.

Methods

Method Summary
Matrix<T> Transform(IEnumerable<string> documents) Transforms documents to SIF sentence embeddings.
void Fit(IEnumerable<string> documents) Fits the SIF vectorizer by computing word frequencies and principal components.