SemanticDeduplicator
Models & Types · AiDotNet.Data.Quality
Detects semantic duplicates using embedding cosine similarity.
How It Works
Semantic deduplication finds documents with the same meaning even if worded differently. Requires pre-computed embeddings (e.g., from a sentence transformer). More expensive than MinHash but catches paraphrased duplicates.
Constructors
| Constructor | Summary |
|---|---|
SemanticDeduplicator(SemanticDeduplicatorOptions options) |
Methods
| Method | Summary |
|---|---|
HashSet<int> FindDuplicates(double[][] embeddings) |
Finds duplicate indices from pre-computed embeddings. |
double CosineSimilarity(double[] a, double[] b) |
Computes cosine similarity between two embedding vectors. |