FetchSGDCompressor<T>

Models & Types · AiDotNet.FederatedLearning.Compression

Implements FetchSGD — Count-Sketch compression with momentum and error accumulation in sketch space.

Inheritance: FederatedLearningComponentBase<T> → FetchSGDCompressor<T>

For Beginners

Each client compresses its gradient into a small count sketch (a fixed-size table that is the same size whatever the model size). Because sketches are linear, the server can add them up, and it keeps momentum and the not-yet-applied error as sketches too, so the server never stores a full-size vector. Each round it recovers the k largest coordinates (“heavy hitters”) from the error sketch, applies them as a sparse update and subtracts their sketch from the error.

How It Works

Algorithm 1 of the paper (per round t, W participating clients):

Reference: Rothchild, D., et al. (2020). “FetchSGD: Communication-Efficient Federated Learning with Sketching.” ICML 2020.

Constructors

Constructor Summary
FetchSGDCompressor(int sketchRows, int sketchCols, int topK, int seed, double momentum, double learningRate) Creates a new FetchSGD compressor.

Properties

Property Summary
double LearningRate { get; } Gets the server learning rate eta.
double Momentum { get; } Gets the server momentum rho.
int SketchCols { get; } Gets the sketch width.
int SketchRows { get; } Gets the sketch dimensions.
int TopK { get; } Gets the top-k value.

Methods

Method Summary
Matrix<T> MergeSketches(IReadOnlyList<Matrix<T>> sketches) Aggregates client sketches: S = (1 / W) * sum_i S_i.
Matrix<T> Sketch(T[] gradient) Client step: compresses a flattened gradient into a count sketch S(g).
T[] RecoverTopK(Matrix<T> mergedSketch, int gradientLength) Recovers the top-k heavy hitters from a sketch without touching the server state.
T[] ServerStep(Matrix<T> aggregatedSketch, int gradientLength) Server step: momentum and error feedback in sketch space, top-k unsketch and error accumulation.
void ResetServerState() Clears the server’s momentum and error sketches.