SIFVectorizer<T>
Models & Types · AiDotNet.Preprocessing.TextVectorizers
Converts text documents to sentence embeddings using Smooth Inverse Frequency (SIF).
Inheritance: TextVectorizerBase<T> → SIFVectorizer<T>
For Beginners
SIF creates sentence embeddings that are surprisingly good:
- Simple: just weighted averaging of word vectors
- Effective: often competitive with complex deep learning methods
- Fast: no neural network inference required
- Requires pre-trained word vectors (Word2Vec, GloVe, etc.)
How It Works
SIF is a simple but effective method for creating sentence embeddings from word vectors. It computes a weighted average of word vectors using smooth inverse frequency weights, then removes the first principal component to create better sentence representations.
The algorithm:
- Compute weighted average: v_s = (1/|s|) * Σ (a / (a + p(w))) * v_w
- Remove first principal component: v_s = v_s - u * u^T * v_s
Constructors
| Constructor | Summary |
|---|---|
SIFVectorizer(Dictionary<string, double[]> wordVectors, double alpha, bool removePrincipalComponent, int nPrincipalComponents, bool lowercase, Func<string, IEnumerable<string>> tokenizer, HashSet<string> stopWords, ITokenizer advancedTokenizer) |
Creates a new instance of SIFVectorizer. |
Properties
| Property | Summary |
|---|---|
bool IsFitted { get; } |
|
int FeatureCount { get; } |
|
int VectorSize { get; } |
Gets the vector dimensionality. |
Methods
| Method | Summary |
|---|---|
Matrix<T> Transform(IEnumerable<string> documents) |
Transforms documents to SIF sentence embeddings. |
void Fit(IEnumerable<string> documents) |
Fits the SIF vectorizer by computing word frequencies and principal components. |