SemanticDeduplicator

Models & Types · AiDotNet.Data.Quality

Detects semantic duplicates using embedding cosine similarity.

How It Works

Semantic deduplication finds documents with the same meaning even if worded differently. Requires pre-computed embeddings (e.g., from a sentence transformer). More expensive than MinHash but catches paraphrased duplicates.

Constructors

Constructor Summary
SemanticDeduplicator(SemanticDeduplicatorOptions options)

Methods

Method Summary
HashSet<int> FindDuplicates(double[][] embeddings) Finds duplicate indices from pre-computed embeddings.
double CosineSimilarity(double[] a, double[] b) Computes cosine similarity between two embedding vectors.