ExactHashDeduplicator
Models & Types · AiDotNet.Data.Quality
Detects exact duplicate documents using cryptographic hashing (SHA-256).
How It Works
The simplest and fastest deduplication method. Uses SHA-256 hashes to find byte-identical documents. Optionally normalizes whitespace and case first.
Constructors
| Constructor | Summary |
|---|---|
ExactHashDeduplicator(ExactHashDeduplicatorOptions options) |
Methods
| Method | Summary |
|---|---|
HashSet<int> FindDuplicates(IReadOnlyList<string> documents) |
Finds duplicate indices from a collection of documents. |
string ComputeHash(string text) |
Computes a hash for the given text. |