MinHashDeduplicator

Models & Types · AiDotNet.Data.Quality

Detects near-duplicate documents using MinHash with Locality-Sensitive Hashing (LSH).

How It Works

MinHash approximates Jaccard similarity between document shingle sets. LSH banding reduces comparison count from O(n^2) to near-linear. Commonly used for deduplicating web crawl data (e.g., C4, The Pile).

Constructors

Constructor Summary
MinHashDeduplicator(MinHashDeduplicatorOptions options)

Methods

Method Summary
HashSet<int> FindDuplicates(IReadOnlyList<string> documents) Finds duplicate indices from a collection of documents using LSH banding.
double EstimateSimilarity(int[] sig1, int[] sig2) Estimates Jaccard similarity between two documents from their MinHash signatures.
int[] ComputeSignature(string text) Computes the MinHash signature for a document.