EmbeddingModelTokenAdapter<T>

Models & Types · AiDotNet.RetrievalAugmentedGeneration.Embeddings

Adapts any real single-vector IEmbeddingModel into a per-token ITokenEmbeddingModel by tokenizing the text and embedding each token through the underlying model. This is what lets ColBERT-style late interaction run over the library’s existing real embedders (Word2Vec / GloVe / FastText / ONNX sentence-transformer / hosted providers) instead of any placeholder — the token vectors are exactly the real model’s output for each token.

Implements: ITokenEmbeddingModel<T>

For Beginners

ColBERT scores a query against a document by comparing them token-by-token, so it needs a vector per word. Most embedding models return a single vector for a whole string. This adapter bridges the two: it splits the text into words and asks the real model to embed each word, giving you genuine per-token embeddings with no fake/placeholder data.

How It Works

Contextual (transformer) token embeddings are strictly stronger than embedding words in isolation; supply a contextual token model directly if you have one. This adapter makes any word/sentence embedder usable, which is the correct real fallback — not a synthetic hash.

Constructors

Constructor Summary
EmbeddingModelTokenAdapter(IEmbeddingModel<T> embeddingModel) Creates a token-embedding adapter over a real single-vector embedding model.

Properties

Property Summary
int EmbeddingDimension { get; }

Methods

Method Summary
Vector<T>[] EmbedTokens(string text)