ExactHashDeduplicator

Models & Types · AiDotNet.Data.Quality

Detects exact duplicate documents using cryptographic hashing (SHA-256).

How It Works

The simplest and fastest deduplication method. Uses SHA-256 hashes to find byte-identical documents. Optionally normalizes whitespace and case first.

Constructors

Constructor Summary
ExactHashDeduplicator(ExactHashDeduplicatorOptions options)

Methods

Method Summary
HashSet<int> FindDuplicates(IReadOnlyList<string> documents) Finds duplicate indices from a collection of documents.
string ComputeHash(string text) Computes a hash for the given text.