TopKSparsificationCompressor<T>

Models & Types · AiDotNet.FederatedLearning.Compression

Implements Top-k Sparsification — send only the k largest gradient elements.

Inheritance: FederatedLearningComponentBase<T> → TopKSparsificationCompressor<T>

For Beginners

Most gradient values are small and contribute little to learning. Top-k sparsification identifies the k largest values (by magnitude) and only sends those, zeroing out the rest. With error feedback (accumulating the “residual” for next round), this converges to the same solution as full gradient but with much less communication.

How It Works

Algorithm:

Reference: Aji, A. & Heafield, K. (2017). “Sparse Communication for Distributed Gradient Descent.” EMNLP 2017.

Constructors

Constructor Summary
TopKSparsificationCompressor(double compressionRatio, bool useErrorFeedback) Creates a new Top-k sparsification compressor.

Properties

Property Summary
bool UseErrorFeedback { get; } Gets whether error feedback is enabled.
double CompressionRatio { get; } Gets the compression ratio (fraction of elements kept).

Methods

Method Summary
Dictionary<string, List<ValueTuple<int, T>>> CompressSparse(Dictionary<string, T[]> gradient, int clientId) Compresses a gradient to a sparse representation (index-value pairs), saving bandwidth.
Dictionary<string, T[]> AggregateSparse(Dictionary<int, Dictionary<string, List<ValueTuple<int, T>>>> clientSparseGradients, Dictionary<string, int> layerSizes) Aggregates sparse gradients from multiple clients by summing overlapping entries.
Dictionary<string, T[]> Compress(Dictionary<string, T[]> gradient, int clientId) Compresses a gradient using top-k sparsification with optional error feedback.
Dictionary<string, T[]> Decompress(Dictionary<string, List<ValueTuple<int, T>>> sparse, Dictionary<string, int> layerSizes) Decompresses a sparse representation back to a dense gradient dictionary.