TopKSparsificationCompressor<T>
Models & Types · AiDotNet.FederatedLearning.Compression
Implements Top-k Sparsification — send only the k largest gradient elements.
Inheritance: FederatedLearningComponentBase<T> → TopKSparsificationCompressor<T>
For Beginners
Most gradient values are small and contribute little to learning. Top-k sparsification identifies the k largest values (by magnitude) and only sends those, zeroing out the rest. With error feedback (accumulating the “residual” for next round), this converges to the same solution as full gradient but with much less communication.
How It Works
Algorithm:
Reference: Aji, A. & Heafield, K. (2017). “Sparse Communication for Distributed Gradient Descent.” EMNLP 2017.
Constructors
| Constructor | Summary |
|---|---|
TopKSparsificationCompressor(double compressionRatio, bool useErrorFeedback) |
Creates a new Top-k sparsification compressor. |
Properties
| Property | Summary |
|---|---|
bool UseErrorFeedback { get; } |
Gets whether error feedback is enabled. |
double CompressionRatio { get; } |
Gets the compression ratio (fraction of elements kept). |
Methods
| Method | Summary |
|---|---|
Dictionary<string, List<ValueTuple<int, T>>> CompressSparse(Dictionary<string, T[]> gradient, int clientId) |
Compresses a gradient to a sparse representation (index-value pairs), saving bandwidth. |
Dictionary<string, T[]> AggregateSparse(Dictionary<int, Dictionary<string, List<ValueTuple<int, T>>>> clientSparseGradients, Dictionary<string, int> layerSizes) |
Aggregates sparse gradients from multiple clients by summing overlapping entries. |
Dictionary<string, T[]> Compress(Dictionary<string, T[]> gradient, int clientId) |
Compresses a gradient using top-k sparsification with optional error feedback. |
Dictionary<string, T[]> Decompress(Dictionary<string, List<ValueTuple<int, T>>> sparse, Dictionary<string, int> layerSizes) |
Decompresses a sparse representation back to a dense gradient dictionary. |