MinHashDeduplicator
Models & Types · AiDotNet.Data.Quality
Detects near-duplicate documents using MinHash with Locality-Sensitive Hashing (LSH).
How It Works
MinHash approximates Jaccard similarity between document shingle sets. LSH banding reduces comparison count from O(n^2) to near-linear. Commonly used for deduplicating web crawl data (e.g., C4, The Pile).
Constructors
| Constructor | Summary |
|---|---|
MinHashDeduplicator(MinHashDeduplicatorOptions options) |
Methods
| Method | Summary |
|---|---|
HashSet<int> FindDuplicates(IReadOnlyList<string> documents) |
Finds duplicate indices from a collection of documents using LSH banding. |
double EstimateSimilarity(int[] sig1, int[] sig2) |
Estimates Jaccard similarity between two documents from their MinHash signatures. |
int[] ComputeSignature(string text) |
Computes the MinHash signature for a document. |