FetchSGDCompressor<T>
Models & Types · AiDotNet.FederatedLearning.Compression
Implements FetchSGD — Count-Sketch compression with momentum and error accumulation in sketch space.
Inheritance: FederatedLearningComponentBase<T> → FetchSGDCompressor<T>
For Beginners
Each client compresses its gradient into a small count sketch (a fixed-size table that is the same size whatever the model size). Because sketches are linear, the server can add them up, and it keeps momentum and the not-yet-applied error as sketches too, so the server never stores a full-size vector. Each round it recovers the k largest coordinates (“heavy hitters”) from the error sketch, applies them as a sparse update and subtracts their sketch from the error.
How It Works
Algorithm 1 of the paper (per round t, W participating clients):
Reference: Rothchild, D., et al. (2020). “FetchSGD: Communication-Efficient Federated Learning with Sketching.” ICML 2020.
Constructors
| Constructor | Summary |
|---|---|
FetchSGDCompressor(int sketchRows, int sketchCols, int topK, int seed, double momentum, double learningRate) |
Creates a new FetchSGD compressor. |
Properties
| Property | Summary |
|---|---|
double LearningRate { get; } |
Gets the server learning rate eta. |
double Momentum { get; } |
Gets the server momentum rho. |
int SketchCols { get; } |
Gets the sketch width. |
int SketchRows { get; } |
Gets the sketch dimensions. |
int TopK { get; } |
Gets the top-k value. |
Methods
| Method | Summary |
|---|---|
Matrix<T> MergeSketches(IReadOnlyList<Matrix<T>> sketches) |
Aggregates client sketches: S = (1 / W) * sum_i S_i. |
Matrix<T> Sketch(T[] gradient) |
Client step: compresses a flattened gradient into a count sketch S(g). |
T[] RecoverTopK(Matrix<T> mergedSketch, int gradientLength) |
Recovers the top-k heavy hitters from a sketch without touching the server state. |
T[] ServerStep(Matrix<T> aggregatedSketch, int gradientLength) |
Server step: momentum and error feedback in sketch space, top-k unsketch and error accumulation. |
void ResetServerState() |
Clears the server’s momentum and error sketches. |